sql: add index_usage_stats and datums_to_bytes to information_schema #156963

angles-n-daemons · 2025-11-05T20:20:30Z

As part of the stable sql api effort, we're aiming to expose all "supported" introspection tools in the information_schema. So that teams have an examplar, we here move index_usage_stats and datums_to_bytes both to information_schema, providing a foundation for future exposure of sql introspection tooling.

Fixes: #156677
Epic: CRDB-56107
Release note (sql change): Duplicates crdb_internal.index_usage_stats and crdb_internal.datums_to_bytes in information_schema (information_schema.crdb_index_usage_stats and
information_schema.crdb_datums_to_bytes respectively).

blathers-crl · 2025-11-05T20:20:34Z

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

cockroach-teamcity · 2025-11-05T20:20:46Z

This change is

michae2

@michae2 reviewed 13 of 17 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @angles-n-daemons)

pkg/sql/crdb_internal.go line 242 at r2 (raw file):

// customer use in production for legacy reasons.
var SupportedCRDBInternalTables = map[string]struct{}{
	// LOCKED: Do not add to this list.

What does "LOCKED" mean?

pkg/sql/sem/builtins/builtins.go line 4751 at r2 (raw file):

		}),

	"crdb_internal.datums_to_bytes":           datumsToBytes,

Is there an allowlist similar to SupportedCRDBInternalTables but for builtin functions? If there were such a list, I would nominate crdb_internal.datums_to_bytes for it, because this function is used in the definition of every existing user-created hash-partitioned index. If we disallow usage of this function by default, I'm not sure applications will be able to write to or read from existing tables with hash-partitioned indexes.

(There are other functions I would also add to such a list, including:

crdb_internal.locality_value
crdb_internal.request_statement_bundle
crdb_internal.clear_query_plan_cache
crdb_internal.clear_statement_hints_cache
crdb_internal.clear_table_stats_cache
crdb_internal.inject_hint
crdb_internal.reset_sql_stats
crdb_internal.reset_index_usage_stats
crdb_internal.set_vmodule
crdb_internal.log
all of which I believe are intended to be available to users.)

yuzefovich

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @angles-n-daemons and @michae2)

pkg/sql/crdb_internal.go line 241 at r2 (raw file):

// SupportedCRDBInternal are the crdb_internal tables that are "supported" for real
// customer use in production for legacy reasons.
var SupportedCRDBInternalTables = map[string]struct{}{

nit: it'd probably be worth adding an automatic check for this by having a unit test that asserts the contents against the hard-coded expectation - this way the CI at least will highlight that no new entries should be added.

angles-n-daemons

@angles-n-daemons reviewed 1 of 17 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @michae2 and @yuzefovich)

pkg/sql/sem/builtins/builtins.go line 4751 at r2 (raw file):

Previously, michae2 (Michael Erickson) wrote…

Is there an allowlist similar to SupportedCRDBInternalTables but for builtin functions? If there were such a list, I would nominate crdb_internal.datums_to_bytes for it, because this function is used in the definition of every existing user-created hash-partitioned index. If we disallow usage of this function by default, I'm not sure applications will be able to write to or read from existing tables with hash-partitioned indexes.

(There are other functions I would also add to such a list, including:

crdb_internal.locality_value

crdb_internal.request_statement_bundle

crdb_internal.clear_query_plan_cache

crdb_internal.clear_statement_hints_cache

crdb_internal.clear_table_stats_cache

crdb_internal.inject_hint

crdb_internal.reset_sql_stats

crdb_internal.reset_index_usage_stats

crdb_internal.set_vmodule

crdb_internal.log
all of which I believe are intended to be available to users.)

Ah, yes we actually created it (and datums_to_bytes is a part of it). I'll check with my team, but so long as these are safe, we'll make sure they make it in too.

angles-n-daemons

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @michae2 and @yuzefovich)

pkg/sql/crdb_internal.go line 241 at r2 (raw file):

Previously, yuzefovich (Yahor Yuzefovich) wrote…

nit: it'd probably be worth adding an automatic check for this by having a unit test that asserts the contents against the hard-coded expectation - this way the CI at least will highlight that no new entries should be added.

that's a great idea, I'll add it to this PR

pkg/sql/crdb_internal.go line 242 at r2 (raw file):

Previously, michae2 (Michael Erickson) wrote…

What does "LOCKED" mean?

Ah, LOCKED just generally means we shouldn't be adding to this list of supported tables in the future, that we should instead be exposing information through the "Supported SQL Interface":

https://docs.google.com/document/d/1STbb8bljTzK_jXRIJrxtijWsPhGErdH1vZdunzPwXvs/edit?tab=t.0

michae2

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @angles-n-daemons and @yuzefovich)

pkg/sql/crdb_internal.go line 242 at r2 (raw file):

Previously, angles-n-daemons (Brian Dillmann) wrote…

Ah, LOCKED just generally means we shouldn't be adding to this list of supported tables in the future, that we should instead be exposing information through the "Supported SQL Interface":

https://docs.google.com/document/d/1STbb8bljTzK_jXRIJrxtijWsPhGErdH1vZdunzPwXvs/edit?tab=t.0

What happens if someone adds to this list?

michae2

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @angles-n-daemons and @yuzefovich)

pkg/sql/crdb_internal.go line 242 at r2 (raw file):

Previously, michae2 (Michael Erickson) wrote…

What happens if someone adds to this list?

(Asking because "LOCKED: Do not add to this list." is not a very clear explanation of why this list exists or why someone shouldn't add to it. What if someone discovers an existing crdb_internal table that a customer is reliant on? Can they add it to this list or not?)

michae2

Aside from my complaining about that comment, this

@michae2 reviewed 3 of 17 files at r1.
Reviewable status: complete! 1 of 0 LGTMs obtained (waiting on @angles-n-daemons and @yuzefovich)

angles-n-daemons

Reviewable status: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @michae2 and @yuzefovich)

pkg/sql/crdb_internal.go line 242 at r2 (raw file):

Previously, michae2 (Michael Erickson) wrote…

(Asking because "LOCKED: Do not add to this list." is not a very clear explanation of why this list exists or why someone shouldn't add to it. What if someone discovers an existing crdb_internal table that a customer is reliant on? Can they add it to this list or not?)

The general idea is we only want to surface reviewed / clearly owned tables for end customers. In the future we want all customer accessible tables to be in the "stable sql interface (information_schema)" which adheres to stricter requirements than crdb_internal.

As part of the stable sql api effort, we're aiming to expose all "supported" internals to the information_schema. So that teams have an examplar, we here move index_usage_stats and datums_to_bytes both to information_schema, providing a foundation for future exposure of system internals. Fixes: cockroachdb#156677 Epic: CRDB-56107 Release note (sql change): Duplicates crdb_internal.index_usage_stats and crdb_internal.datums_to_bytes in information_schema (information_schema.crdb_index_usage_stats and information_schema.crdb_datums_to_bytes respectively).

As part of the stable SQL API effort, expose the following builtin functions in information_schema to complement the existing crdb_internal versions: - crdb_internal.reset_index_usage_stats → information_schema.crdb_reset_index_usage_stats - crdb_internal.reset_sql_stats → information_schema.crdb_reset_sql_stats - crdb_internal.clear_query_plan_cache → information_schema.crdb_clear_query_plan_cache - crdb_internal.clear_table_stats_cache → information_schema.crdb_clear_table_stats_cache - crdb_internal.clear_statement_hints_cache → information_schema.crdb_clear_statement_hints_cache The implementations are shared between both schemas to ensure identical behavior. All existing test usages throughout the codebase have been updated to use the information_schema versions to demonstrate the new functions work correctly. This follows the pattern established in PR cockroachdb#156963 which ported index_usage_statistics and datums_to_bytes to information_schema. Epic: None Release note (sql change): Added information_schema versions of 5 builtin functions: crdb_reset_index_usage_stats, crdb_reset_sql_stats, crdb_clear_query_plan_cache, crdb_clear_table_stats_cache, and crdb_clear_statement_hints_cache. These complement the existing crdb_internal versions as part of the stable SQL API effort. Co-Authored-By: Claude <noreply@anthropic.com>

angles-n-daemons · 2025-12-08T20:25:37Z

bors r+

156963: sql: add index_usage_stats and datums_to_bytes to information_schema r=angles-n-daemons a=angles-n-daemons As part of the stable sql api effort, we're aiming to expose all "supported" introspection tools in the information_schema. So that teams have an examplar, we here move index_usage_stats and datums_to_bytes both to information_schema, providing a foundation for future exposure of sql introspection tooling. Fixes: #156677 Epic: CRDB-56107 Release note (sql change): Duplicates crdb_internal.index_usage_stats and crdb_internal.datums_to_bytes in information_schema (information_schema.crdb_index_usage_stats and information_schema.crdb_datums_to_bytes respectively). 159004: logictest: deflake TestReadCommittedLogic_cluster_locks r=miraradeva a=arulajmani The test was trying to construct a scenario where there were two requests waiting on a single key. The test expected the order in which requests arrived to be deterministic, as the lock table releases requests in FIFO order. This isn't true with the async statement framework in logic tests. Here, we rework the test to not need two waiters on a single key; instead, we perform each test separately, resetting the test state in between runs. Closes #158769 Release note: None Co-authored-by: Brian Dillmann <brian.dillmann@cockroachlabs.com> Co-authored-by: Arul Ajmani <arulajmani@gmail.com>

As part of the stable SQL API effort, expose the following builtin functions in information_schema to complement the existing crdb_internal versions: - crdb_internal.reset_index_usage_stats → information_schema.crdb_reset_index_usage_stats - crdb_internal.reset_sql_stats → information_schema.crdb_reset_sql_stats - crdb_internal.clear_query_plan_cache → information_schema.crdb_clear_query_plan_cache - crdb_internal.clear_table_stats_cache → information_schema.crdb_clear_table_stats_cache - crdb_internal.clear_statement_hints_cache → information_schema.crdb_clear_statement_hints_cache The implementations are shared between both schemas to ensure identical behavior. All existing test usages throughout the codebase have been updated to use the information_schema versions to demonstrate the new functions work correctly. This follows the pattern established in PR cockroachdb#156963 which ported index_usage_statistics and datums_to_bytes to information_schema. Epic: None Release note (sql change): Added information_schema versions of 5 builtin functions: crdb_reset_index_usage_stats, crdb_reset_sql_stats, crdb_clear_query_plan_cache, crdb_clear_table_stats_cache, and crdb_clear_statement_hints_cache. These complement the existing crdb_internal versions as part of the stable SQL API effort. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

craig · 2025-12-08T21:03:51Z

This PR was included in a batch that was canceled, it will be automatically retried

craig · 2025-12-08T22:00:17Z

Build succeeded:

As part of the stable SQL API effort, expose the following builtin functions in information_schema to complement the existing crdb_internal versions: - crdb_internal.reset_index_usage_stats → information_schema.crdb_reset_index_usage_stats - crdb_internal.reset_sql_stats → information_schema.crdb_reset_sql_stats - crdb_internal.clear_query_plan_cache → information_schema.crdb_clear_query_plan_cache - crdb_internal.clear_table_stats_cache → information_schema.crdb_clear_table_stats_cache - crdb_internal.clear_statement_hints_cache → information_schema.crdb_clear_statement_hints_cache The implementations are shared between both schemas to ensure identical behavior. All existing test usages throughout the codebase have been updated to use the information_schema versions to demonstrate the new functions work correctly. This follows the pattern established in PR cockroachdb#156963 which ported index_usage_statistics and datums_to_bytes to information_schema. Epic: None Release note (sql change): Added information_schema versions of 5 builtin functions: crdb_reset_index_usage_stats, crdb_reset_sql_stats, crdb_clear_query_plan_cache, crdb_clear_table_stats_cache, and crdb_clear_statement_hints_cache. These complement the existing crdb_internal versions as part of the stable SQL API effort. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

angles-n-daemons marked this pull request as ready for review November 5, 2025 20:22

angles-n-daemons requested a review from a team as a code owner November 5, 2025 20:22

angles-n-daemons force-pushed the stable-index-usage-stats branch from a433c76 to ffe82c9 Compare December 1, 2025 21:37

angles-n-daemons requested a review from a team as a code owner December 1, 2025 21:37

angles-n-daemons requested review from michae2 and removed request for a team December 1, 2025 21:37

angles-n-daemons force-pushed the stable-index-usage-stats branch 2 times, most recently from 14a8b98 to 18ebd8d Compare December 2, 2025 19:02

michae2 reviewed Dec 3, 2025

View reviewed changes

yuzefovich reviewed Dec 3, 2025

View reviewed changes

angles-n-daemons commented Dec 3, 2025

View reviewed changes

michae2 reviewed Dec 3, 2025

View reviewed changes

michae2 approved these changes Dec 4, 2025

View reviewed changes

angles-n-daemons force-pushed the stable-index-usage-stats branch 2 times, most recently from 3d55494 to c266da3 Compare December 4, 2025 18:49

angles-n-daemons commented Dec 4, 2025

View reviewed changes

angles-n-daemons force-pushed the stable-index-usage-stats branch from c266da3 to d40d0ca Compare December 5, 2025 17:17

angles-n-daemons force-pushed the stable-index-usage-stats branch from d40d0ca to 470aff4 Compare December 5, 2025 18:14

michae2 mentioned this pull request Dec 6, 2025

sql: port 5 builtin functions to information_schema #158938

Open

craig bot merged commit 854a5cf into cockroachdb:master Dec 8, 2025
33 of 34 checks passed

celeste-cockroachdb bot added the target-release-26.1.0 label Dec 8, 2025

sql: add index_usage_stats and datums_to_bytes to information_schema #156963

sql: add index_usage_stats and datums_to_bytes to information_schema #156963

Uh oh!

Conversation

angles-n-daemons commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blathers-crl bot commented Nov 5, 2025

Uh oh!

cockroach-teamcity commented Nov 5, 2025

Uh oh!

michae2 left a comment

Choose a reason for hiding this comment

Uh oh!

yuzefovich left a comment

Choose a reason for hiding this comment

Uh oh!

angles-n-daemons left a comment

Choose a reason for hiding this comment

Uh oh!

angles-n-daemons left a comment

Choose a reason for hiding this comment

Uh oh!

michae2 left a comment

Choose a reason for hiding this comment

Uh oh!

michae2 left a comment

Choose a reason for hiding this comment

Uh oh!

michae2 left a comment

Choose a reason for hiding this comment

Uh oh!

angles-n-daemons left a comment

Choose a reason for hiding this comment

Uh oh!

angles-n-daemons commented Dec 8, 2025

Uh oh!

craig bot commented Dec 8, 2025

Uh oh!

craig bot commented Dec 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

angles-n-daemons commented Nov 5, 2025 •

edited

Loading