sql: enabling forward indexes and ORDERBY on JSONB columns #99275

Shivs11 · 2023-03-22T20:01:01Z

Currently, #97928 outlines the scheme for JSONB encoding
and decoding for forward indexes. However, the PR doesn't
enable this feature to our users. This current PR aims
to allow forward indexes on JSONB columns. The presence
of a lexicographical ordering, as described in #97928,
shall now allow primary and secondary indexes on JSONB
columns along with the ability to use ORDER BY filter
in their queries.

Additionally, JSON values consist of decimal numbers
and containers, such as Arrays and Objects, which can
contain these decimal numbers. In order to preserve
the values after the decimal, JSONB columns are now
required to be composite in nature. This shall enable
such values to be stored in both the key and the value
side of a K/V pair in hopes of receiving the exact value.

Fixes: #35706

Release note (sql change): This PR adds support for enabling
forward indexes and ordering on JSON values.

Epic: CRDB-24501

cockroach-teamcity · 2023-03-22T20:01:12Z

This change is

Shivs11

Reviewable status: complete! 0 of 0 LGTMs obtained

pkg/sql/ttl/ttljob/ttljob_test.go line 583 at r2 (raw file):

			indexableTyps = append(indexableTyps, typ)
		}
	}

An issue has been raised here: #99432

Previously, cockroachdb#99275 and cockroachdb#97298 were responsible for laying the foundations of a JSON lexicographical order as well as enabling JSON forward indexes in CRDB. However, these commits did not account for version gates. In a given cluster, all of its nodes are required to be at a certain version number for creating JSON forward indexes as well as for ordering JSON columns. Thus, this PR aims to enable version gates for ensuring all nodes in a given cluster meet this requirement. Epic: CRDB-24501 Release note (sql change): This PR adds version gates which requires all nodes in a given cluster to have a minimum binary version number which in turn is required for creating forward indexes on JSON columns and for ordering JSON columns. Fixes: cockroachdb#35706

Previously, cockroachdb#99275 and cockroachdb#97928 were responsible for laying the foundations of a JSON lexicographical order as well as enabling JSON forward indexes in CRDB. However, these commits did not account for version gates. In a given cluster, all of its nodes are required to be at a certain version number for creating JSON forward indexes as well as for ordering JSON columns. Thus, this PR aims to enable version gates for ensuring all nodes in a given cluster meet this requirement. Epic: CRDB-24501 Fixes: cockroachdb#35706 Release note (sql change): This PR adds version gates which requires all nodes in a given cluster to have a minimum binary version number which in turn is required for creating forward indexes on JSON columns and for ordering JSON columns.

101932: sql: adding version gates for changes related to JSON forward indexes r=fqazi a=Shivs11 Previously, #99275 and #97928 were responsible for laying the foundations of a JSON lexicographical order as well as enabling JSON forward indexes in CRDB. However, these commits did not account for version gates. In a given cluster, all of its nodes are required to be at a certain version number for creating JSON forward indexes as well as for ordering JSON columns. Thus, this PR aims to enable version gates for ensuring all nodes in a given cluster meet this requirement. Epic: [CRDB-24501](https://cockroachlabs.atlassian.net/browse/CRDB-24501) Fixes: #35706 Release note (sql change): This PR adds version gates which requires all nodes in a given cluster to have a minimum binary version number which in turn is required for creating forward indexes on JSON columns and for ordering JSON columns. Co-authored-by: Shivam Saraf <shivam.saraf@cockroachlabs.com>

Currently, cockroachdb#97928 and cockroachdb#99275 are responsible for laying out a lexicographical ordering for JSON columns to be forward indexable in nature. This ordering is based on the rules posted by Postgres and is in cockroachdb#99849. However, Postgres currently sorts the empty JSON array before any other JSON values. A Postgres bug report for this has been opened: https://www.postgresql.org/message-id/17873-826fdc8bbcace4f1%40postgresql.org This PR intends on replicating the Postgres behavior. Fixes cockroachdb#105668 Epic: CRDB-24501 Release note: None

101260: sql: replicating JSON empty array ordering found in Postgres r=mgartner a=Shivs11 Currently, #97928 and #99275 are responsible for laying out a lexicographical ordering for JSON columns to be forward indexable in nature. This ordering is based on the rules posted by Postgres and is in #99849. However, Postgres currently sorts the empty JSON array before any other JSON values. A Postgres bug report for this has been opened: https://www.postgresql.org/message-id/17873-826fdc8bbcace4f1%40postgresql.org This PR intends on replicating the Postgres behavior. Fixes #105668 Epic: CRDB-24501 Release note: None 108160: roachtest/awsdms: run once a week instead r=Jeremyyang920 a=otan Save a bit of mad dosh by running awsdms once a weekly instead of daily. We don't need this tested every week. Epic: None Release note: None 108300: schemachanger: Unskip some backup tests r=Xiang-Gu a=Xiang-Gu Randomly skip subtests in the BACKUP/RESTORE suites before parallelizing them. Epic: None Release note: None 108328: rowexec: fix TestUncertaintyErrorIsReturned under race r=yuzefovich a=yuzefovich We just saw a case when `TestUncertaintyErrorIsReturned` failed under race because we got a different DistSQL plan. This seems plausible in case the range cache population (which the test does explicitly) isn't quick enough for some reason, so this commit allows for the DistSQL plan to match the expectation via `SucceedsSoon` (if we happen to get a bad plan, then the following query execution should have the up-to-date range cache). Fixes: #108250. Release note: None 108341: importer: fix stale comment on mysqlStrToDatum r=mgartner,DrewKimball a=otan Release note: None Epic: None From #108286 (review) 108370: go.mod: bump Pebble to fffe02a195e3 r=RahulAggarwal1016 a=RahulAggarwal1016 fffe02a1 db: simplify ScanInternal() df7e2ae1 vfs: deflake TestDiskHealthChecking_Filesystem ff5c929a Rate Limit Scan Statistics af8c5f27 internal/cache: mark panic messages as redaction-safe Epic: none Release note: none 108379: changefeedccl: deflake TestChangefeedSchemaChangeBackfillCheckpoint r=miretskiy a=jayshrivastava Previously, the test `TestChangefeedSchemaChangeBackfillCheckpoint` would fail because it would read a table span too early. A schema change using the delcarative schema changer will update a table span to point to a new set of ranges. Previously, this test would use the span from before the schema change, which is incorrect. This change makes it use the span from after the schema change. I stress tested this 30k times under the new schema changer and 10k times under the legacy schema changer to ensure the test is not flaky anymore. Fixes: #108084 Release note: None Epic: None Co-authored-by: Shivam Saraf <shivam.saraf@cockroachlabs.com> Co-authored-by: Oliver Tan <otan@cockroachlabs.com> Co-authored-by: Xiang Gu <xiang@cockroachlabs.com> Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com> Co-authored-by: Rahul Aggarwal <rahul.aggarwal@cockroachlabs.com> Co-authored-by: Jayant Shrivastava <jayants@cockroachlabs.com>

This commit adds a regression test for a bug that was unknowingly fixed in cockroachdb#99275. See cockroachdb#113103 for more details. Release note: None

113106: sql: add regression test for previously fixed JSON bug r=mgartner a=mgartner #### opt: remove unnecessary argument in memo.CanBeCompositeSensitive Release note: None #### sql: add regression test for previously fixed JSON bug This commit adds a regression test for a bug that was unknowingly fixed in #99275. See #113103 for more details. Epic: None Release note: None Co-authored-by: Marcus Gartner <marcus@cockroachlabs.com>

Shivs11 force-pushed the jsonb_forward_indexes_composite_testing branch 3 times, most recently from 1b476b6 to 09a0281 Compare March 28, 2023 21:42

Shivs11 changed the title ~~Jsonb forward indexes composite testing~~ sql: enabling forward indexes and ORDERBY on JSONB columns Mar 28, 2023

Shivs11 force-pushed the jsonb_forward_indexes_composite_testing branch 13 times, most recently from dbbdd2b to e14460d Compare March 29, 2023 20:59

Shivs11 commented Mar 30, 2023

View reviewed changes

Shivs11 force-pushed the jsonb_forward_indexes_composite_testing branch 4 times, most recently from 44fc0db to 890ba2f Compare March 30, 2023 20:00

Shivs11 marked this pull request as ready for review March 30, 2023 20:07

Shivs11 requested review from a team March 30, 2023 20:07

Shivs11 requested review from a team as code owners March 30, 2023 20:07

Shivs11 requested review from rharding6373 and removed request for rharding6373 March 30, 2023 20:07

rafiss mentioned this pull request Jun 27, 2023

sql: decide on JSON ordering before 23.2 release #105668

Closed

mgartner mentioned this pull request Oct 3, 2023

sql: implement json compare #43123

Closed

mgartner mentioned this pull request Oct 25, 2023

sql: add regression test for previously fixed JSON bug #113106

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sql: enabling forward indexes and ORDERBY on JSONB columns #99275

sql: enabling forward indexes and ORDERBY on JSONB columns #99275

Shivs11 commented Mar 22, 2023 •

edited

cockroach-teamcity commented Mar 22, 2023

Shivs11 left a comment

sql: enabling forward indexes and ORDERBY on JSONB columns #99275

sql: enabling forward indexes and ORDERBY on JSONB columns #99275

Conversation

Shivs11 commented Mar 22, 2023 • edited

cockroach-teamcity commented Mar 22, 2023

Shivs11 left a comment

Choose a reason for hiding this comment

Shivs11 commented Mar 22, 2023 •

edited