sql: do not hide columns in introspection tables

[knz]

Fixes #28548.

Requested/suggested by @bdarnell

Clients were confused by the fact some items in introspection referred
to columns that were otherwise not listed because they were marked as
hidden in the table descriptors. For example, an index would
refer to a column (rowid) not listed in the columns for that table.

This has been found (by Ben) to be a problem for SQLAlchemy and (by
me) in dBeaver.

In contrast, PostgreSQL which also has hidden columns always shows all
columns in introspection, including hidden columns, in both
information_schema and pg_catalog.

This patch corrects the issue by doing the same as PostgreSQL. The
output of SHOW COLUMNS is extended to indicate which columns are
hidden.

Release note (bug fix): Hidden columns are now listed in
information_schema and pg_catalog tables, for better compatibility
with PostgreSQL.

Release note (sql change): The output of SHOW COLUMNS is extended to
indicate which columns are hidden.

[cockroach-teamcity]

This change is 

[jordanlewis]

Yikes, this looks like it breaks one of the example orms checks...

Reviewable status: complete! 0 of 0 LGTMs obtained

[jordanlewis]

LGTM mod comment, thanks!

[jordanlewis]

I have a medium-strength objection to this. It's trouble to add things to information_schema unless we absolutely have to. I would prefer to leave information_schema.columns the same, and let the hidden-ness be ambiguous, just like it is in Postgres.

[knz]

The failing test was because example-orms needed a boost (you approved that PR). The tests should pass now.

I do not understand your concern about information_schema. Everyone seems to place their extensions in here:

• mysql generation_expression etc: https://dev.mysql.com/doc/refman/5.7/en/columns-table.html
• mssql domain_catalog etc: https://www.mssqltips.com/sqlservertutorial/183/informationschemacolumns/
• pg udt_catalog etc: https://www.postgresql.org/docs/10/static/infoschema-columns.html

What's the trouble exactly?

[jordanlewis]

My concern is that we're adding pointless additional complexity to the table, which puts additional backward compatibility burden on us. This is pretty low stakes, but I don't want to establish a precedent of adding columns to information_schema unless for some reason we really have to. Postgres doesn't have a hidden column in this, even though they have hidden columns in all their tables (oid, xid, etc) - why should we?

[knz]

Postgres doesn't have a hidden column in this, even though they have hidden columns in all their tables (oid, xid, etc) - why should we?

Because cockroach dump needs a way to exclude hidden columns when it inspects the schema of a table to dump. If you can find a better way to exclude them I'll take it. If you're really set against it, I'll simply create a new crdb_internal table and call it a day.

[jordanlewis]

I didn't realize you actually needed the column. In that case, I think I'm fine with it. LGTM!

[knz]

thank you! 💖

bors r+

[craig]

Build succeeded

• GitHub CI (Cockroach)