Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement INFORMATION_SCHEMA.USER_DEFINED_TYPES and INFORMATION_SCHEMA.ATTRIBUTES #109603

Closed
lukaseder opened this issue Aug 28, 2023 · 4 comments · Fixed by #111401
Closed

Implement INFORMATION_SCHEMA.USER_DEFINED_TYPES and INFORMATION_SCHEMA.ATTRIBUTES #109603

lukaseder opened this issue Aug 28, 2023 · 4 comments · Fixed by #111401
Assignees
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-community Originated from the community T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-blathers-triaged blathers was able to find an owner

Comments

@lukaseder
Copy link

lukaseder commented Aug 28, 2023

Is your feature request related to a problem? Please describe.

I would like to reverse engineer user-defined types from the INFORMATION_SCHEMA via the standard views:

  • USER_DEFINED_TYPES
  • ATTRIBUTES

Describe alternatives you've considered

I can probably work with queries against pg_type and pg_attribute as I've done before with the lack of ROUTINES support (see #104083):

select *
from pg_catalog.pg_type as t
join pg_catalog.pg_attribute as a on a.attrelid = t.typrelid
where t.typname = 't';

But for the future support of third party tooling, I still think these INFORMATION_SCHEMA views would be useful.

Jira issue: CRDB-31015

@lukaseder lukaseder added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Aug 28, 2023
@blathers-crl
Copy link

blathers-crl bot commented Aug 28, 2023

Hello, I am Blathers. I am here to help you get the issue triaged.

I have CC'd a few people who may be able to assist you:

  • @cockroachdb/sql-foundations (found keywords: pg_,ORM)

If we have not gotten back to your issue within a few business days, you can try the following:

  • Join our community slack channel and ask on #cockroachdb.
  • Try find someone from here if you know they worked closely on the area and CC them.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added O-community Originated from the community X-blathers-triaged blathers was able to find an owner labels Aug 28, 2023
@lukaseder
Copy link
Author

I don't recall what I may have done differently yesterday, but this query no longer seems to produce results:

select *
from pg_catalog.pg_type as t
join pg_catalog.pg_attribute as a on a.attrelid = t.typrelid
where t.typname = 't';

In fact:

create type u as (i int, j int);
select * from pg_attribute as a where attname in ('i', 'j');

Doesn't yield results on version 23.1.8. That means I won't be able to reverse engineer UDTs in any way, I think?

@lukaseder
Copy link
Author

I've created another issue for the pg_class and pg_attribute population, as that is more important, I think:

@rafiss rafiss added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Aug 29, 2023
@blathers-crl blathers-crl bot added this to Triage in SQL Foundations Aug 29, 2023
@rafiss
Copy link
Collaborator

rafiss commented Aug 29, 2023

We should be able to add these as views that match the PG definitions: https://github.com/postgres/postgres/blob/9b581c53418666205938311ef86047aa3c6b741f/src/backend/catalog/information_schema.sql#L2461

Similar to what we did for information_schema.routines: #105944

CREATE VIEW user_defined_types AS
    SELECT CAST(current_database() AS sql_identifier) AS user_defined_type_catalog,
           CAST(n.nspname AS sql_identifier) AS user_defined_type_schema,
           CAST(c.relname AS sql_identifier) AS user_defined_type_name,
           CAST('STRUCTURED' AS character_data) AS user_defined_type_category,
           CAST('YES' AS yes_or_no) AS is_instantiable,
           CAST(null AS yes_or_no) AS is_final,
           CAST(null AS character_data) AS ordering_form,
           CAST(null AS character_data) AS ordering_category,
           CAST(null AS sql_identifier) AS ordering_routine_catalog,
           CAST(null AS sql_identifier) AS ordering_routine_schema,
           CAST(null AS sql_identifier) AS ordering_routine_name,
           CAST(null AS character_data) AS reference_type,
           CAST(null AS character_data) AS data_type,
           CAST(null AS cardinal_number) AS character_maximum_length,
           CAST(null AS cardinal_number) AS character_octet_length,
           CAST(null AS sql_identifier) AS character_set_catalog,
           CAST(null AS sql_identifier) AS character_set_schema,
           CAST(null AS sql_identifier) AS character_set_name,
           CAST(null AS sql_identifier) AS collation_catalog,
           CAST(null AS sql_identifier) AS collation_schema,
           CAST(null AS sql_identifier) AS collation_name,
           CAST(null AS cardinal_number) AS numeric_precision,
           CAST(null AS cardinal_number) AS numeric_precision_radix,
           CAST(null AS cardinal_number) AS numeric_scale,
           CAST(null AS cardinal_number) AS datetime_precision,
           CAST(null AS character_data) AS interval_type,
           CAST(null AS cardinal_number) AS interval_precision,
           CAST(null AS sql_identifier) AS source_dtd_identifier,
           CAST(null AS sql_identifier) AS ref_dtd_identifier

    FROM pg_namespace n, pg_class c, pg_type t

    WHERE n.oid = c.relnamespace
          AND t.typrelid = c.oid
          AND c.relkind = 'c'
          AND (pg_has_role(t.typowner, 'USAGE')
               OR has_type_privilege(t.oid, 'USAGE'));
CREATE VIEW attributes AS
    SELECT CAST(current_database() AS sql_identifier) AS udt_catalog,
           CAST(nc.nspname AS sql_identifier) AS udt_schema,
           CAST(c.relname AS sql_identifier) AS udt_name,
           CAST(a.attname AS sql_identifier) AS attribute_name,
           CAST(a.attnum AS cardinal_number) AS ordinal_position,
           CAST(pg_get_expr(ad.adbin, ad.adrelid) AS character_data) AS attribute_default,
           CAST(CASE WHEN a.attnotnull OR (t.typtype = 'd' AND t.typnotnull) THEN 'NO' ELSE 'YES' END
             AS yes_or_no)
             AS is_nullable, -- This column was apparently removed between SQL:2003 and SQL:2008.

           CAST(
             CASE WHEN t.typelem <> 0 AND t.typlen = -1 THEN 'ARRAY'
                  WHEN nt.nspname = 'pg_catalog' THEN format_type(a.atttypid, null)
                  ELSE 'USER-DEFINED' END
             AS character_data)
             AS data_type,

           CAST(
             _pg_char_max_length(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS cardinal_number)
             AS character_maximum_length,

           CAST(
             _pg_char_octet_length(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS cardinal_number)
             AS character_octet_length,

           CAST(null AS sql_identifier) AS character_set_catalog,
           CAST(null AS sql_identifier) AS character_set_schema,
           CAST(null AS sql_identifier) AS character_set_name,

           CAST(CASE WHEN nco.nspname IS NOT NULL THEN current_database() END AS sql_identifier) AS collation_catalog,
           CAST(nco.nspname AS sql_identifier) AS collation_schema,
           CAST(co.collname AS sql_identifier) AS collation_name,

           CAST(
             _pg_numeric_precision(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS cardinal_number)
             AS numeric_precision,

           CAST(
             _pg_numeric_precision_radix(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS cardinal_number)
             AS numeric_precision_radix,

           CAST(
             _pg_numeric_scale(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS cardinal_number)
             AS numeric_scale,

           CAST(
             _pg_datetime_precision(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS cardinal_number)
             AS datetime_precision,

           CAST(
             _pg_interval_type(_pg_truetypid(a, t), _pg_truetypmod(a, t))
             AS character_data)
             AS interval_type,
           CAST(null AS cardinal_number) AS interval_precision,

           CAST(current_database() AS sql_identifier) AS attribute_udt_catalog,
           CAST(nt.nspname AS sql_identifier) AS attribute_udt_schema,
           CAST(t.typname AS sql_identifier) AS attribute_udt_name,

           CAST(null AS sql_identifier) AS scope_catalog,
           CAST(null AS sql_identifier) AS scope_schema,
           CAST(null AS sql_identifier) AS scope_name,

           CAST(null AS cardinal_number) AS maximum_cardinality,
           CAST(a.attnum AS sql_identifier) AS dtd_identifier,
           CAST('NO' AS yes_or_no) AS is_derived_reference_attribute

    FROM (pg_attribute a LEFT JOIN pg_attrdef ad ON attrelid = adrelid AND attnum = adnum)
         JOIN (pg_class c JOIN pg_namespace nc ON (c.relnamespace = nc.oid)) ON a.attrelid = c.oid
         JOIN (pg_type t JOIN pg_namespace nt ON (t.typnamespace = nt.oid)) ON a.atttypid = t.oid
         LEFT JOIN (pg_collation co JOIN pg_namespace nco ON (co.collnamespace = nco.oid))
           ON a.attcollation = co.oid AND (nco.nspname, co.collname) <> ('pg_catalog', 'default')

    WHERE a.attnum > 0 AND NOT a.attisdropped
          AND c.relkind IN ('c')
          AND (pg_has_role(c.relowner, 'USAGE')
               OR has_type_privilege(c.reltype, 'USAGE'));

@rafiss rafiss assigned annrpom and unassigned rafiss Aug 29, 2023
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 24, 2023
This commit adds an implementation for the
`information_schema._pg_char_octet_length` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `information_schema._pg_char_octet_length`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 24, 2023
This commit adds an implementation for the
`information_schema._pg_datetime_precision` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The
`information_schema._pg_datetime_precision` builtin function is
now supported, which improves compatibility with PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 24, 2023
This commit adds an implementation for the
`information_schema._pg_interval_type` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `information_schema._pg_interval_type`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 24, 2023
This commit adds an implementation for the
`pg_encoding_max_length` builtin.

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `pg_encoding_max_length`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 25, 2023
This patch populates information_schema.user_defined_types with
information about user defined types and information_schema.attributes
with information about the attributes of composite data types.

Epic: none
Fixes: cockroachdb#109603

Release note (sql change): This patch populates
information_schema.user_defined_types with information about user
defined types and information_schema.attributes with information about
the attributes of composite data types.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 31, 2023
This commit adds an implementation for the
`information_schema._pg_char_octet_length` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `information_schema._pg_char_octet_length`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 31, 2023
This commit adds an implementation for the
`pg_encoding_max_length` builtin.

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `pg_encoding_max_length`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 31, 2023
This commit adds an implementation for the
`information_schema._pg_datetime_precision` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The
`information_schema._pg_datetime_precision` builtin function is
now supported, which improves compatibility with PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 31, 2023
This commit adds an implementation for the
`information_schema._pg_interval_type` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `information_schema._pg_interval_type`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Oct 31, 2023
This patch populates information_schema.user_defined_types with
information about user defined types and information_schema.attributes
with information about the attributes of composite data types.

Epic: none
Fixes: cockroachdb#109603

Release note (sql change): This patch populates
information_schema.user_defined_types with information about user
defined types and information_schema.attributes with information about
the attributes of composite data types.
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 6, 2023
This commit adds an implementation for the
`information_schema._pg_char_octet_length` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `information_schema._pg_char_octet_length`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 6, 2023
This commit adds an implementation for the
`pg_encoding_max_length` builtin.

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `pg_encoding_max_length`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 6, 2023
This commit adds an implementation for the
`information_schema._pg_datetime_precision` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The
`information_schema._pg_datetime_precision` builtin function is
now supported, which improves compatibility with PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 6, 2023
This commit adds an implementation for the
`information_schema._pg_interval_type` builtin.

The builtin is implemented as a user-defined function in Postgres
[here](https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql)

Needed for: cockroachdb#109603
Epic: none

Release note (sql change): The `information_schema._pg_interval_type`
builtin function is now supported, which improves compatibility with
PostgreSQL.
annrpom added a commit to annrpom/cockroach that referenced this issue Nov 6, 2023
This patch populates information_schema.user_defined_types with
information about user defined types and information_schema.attributes
with information about the attributes of composite data types.

Epic: none
Fixes: cockroachdb#109603

Release note (sql change): This patch populates
information_schema.user_defined_types with information about user
defined types and information_schema.attributes with information about
the attributes of composite data types.
craig bot pushed a commit that referenced this issue Nov 6, 2023
111401: sql: populate information_schema.user_defined_types and attributes r=annrpom a=annrpom

commit 1-4:
### sql: add necessary builtins for information_schema.user_defined_types 

These patches add the following builtins:
- information_schema._pg_char_octet_length: returns the octet length of a given char/varchar/string
- pg_encoding_max_length: returns the maximum length of a given encoding. Since we only support UTF8, this will return length 4.
- information_schema._pg_datetime_precision: returns the precision of the given datetime-related type
- information_schema._pg_interval_type: displays the interval type duration if it is set

Needed for: #109603
Epic: none

Release note (sql change): The information_schema._pg_char_octet_length, information_schema._pg_datetime_precision, information_schema._pg_interval_type, and pg_encoding_max_length builtin functions are now supported, which improves compatibility with PostgreSQL.

---
last commit:
### sql: populate information_schema.user_defined_types and attributes

This patch populates information_schema.user_defined_types with
information about user defined types and information_schema.attributes
with information about the attributes of composite data types.

Epic: none
Fixes: #109603

Release note (sql change): This patch populates
information_schema.user_defined_types with information about user
defined types and information_schema.attributes with information about
the attributes of composite data types.

113498: obsservice: export StatementInsightsStatistics r=maryliag a=maryliag

_commit 1:_

**obsservice: add stmt insights proto and conversion**

Add proto definition for Statement Insights to be
used on Obs Service.
Creates function to copy a `insights.Insight` to the new
`obspb.StatementInsightsStatistics` format.

For this version, there are some parameters not being set,
such as contention. Those will be populated on a following
iteration.

Epic: none

Release note: None

----

_commit 2:_

**obsservice: export StatementInsightsStatistics**

This patch hooks into the Flush functionality used
by PersistedSQLStats, in preparation for Insights
to be sent to external o11y systems.
A following PR will do the actual export during the flush.

Lastly, it's acknowledged that the transformation
required here is likely going to be heavy on
allocations. During the prototyping phase however,
we leave the optimization and/or restructuring of
the exported type for the future. For now, we use
a sync.Pool in an effort to reduce allocations/GC.

Release note: none


113626: debug: fix help text for `--include-range-info` debug zip flag r=dhartunian a=abarganier

Fixes: #112053

This patch fixes the help text for the `--include-range-info` command available in `cockroach debug zip`. This is necessary, because `debug zip` now only creates a single file per node. The help text was not updated to reflect this.

Release note: none

113835: team: embed TEAMS.yaml r=RaduBerinde a=RaduBerinde

The `internal/team` code looks for `TEAMS.yaml` in the repo. This
means that `roachtest` must be run in the tree (and if the branches
don't match, obscure errors could happen in principle).

This commit moves to embedding the data using `go:embed`. Because
`go:embed` doesn't allow embedding of files outside the package, we
also have to add a generation rule to copy the file.

Fixes: #111661
Release note: None

Co-authored-by: Annie Pompa <annie@cockroachlabs.com>
Co-authored-by: maryliag <marylia@cockroachlabs.com>
Co-authored-by: Alex Barganier <abarganier@cockroachlabs.com>
Co-authored-by: Radu Berinde <radu@cockroachlabs.com>
@craig craig bot closed this as completed in 391f96b Nov 6, 2023
SQL Foundations automation moved this from Triage to Done Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) O-community Originated from the community T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) X-blathers-triaged blathers was able to find an owner
Projects
Development

Successfully merging a pull request may close this issue.

3 participants