staging-v24.3.31: sql: fix partial index data loss / phantom rows during update#168474
Merged
rail merged 1 commit intocockroachdb:staging-v24.3.31from Apr 16, 2026
Merged
Conversation
This commit fixes a bug on tables with multiple column families where a concurrent update that does not overlap with a partial index's column family could cause the partial index to write a NULL instead of the actual data, or incorrectly add phantom rows to a temporary index during a schema change backfill. This bug was previously masked on default (single-column-family) tables because an update to any column causes the optimizer to conservatively fetch all columns in that family. However, with multiple column families, two normalization rules in the optimizer caused issues: 1. PruneMutationFetchCols: If an update does not change any column associated with an index, the optimizer avoids fetching those columns. This causes the execution layer to see NULLs for the unfetched columns. 2. SimplifyPartialIndexProjections: If an update does not change any column associated with a partial index, the optimizer simplifies the partial index predicate evaluation to FALSE. This causes the execution layer to skip writes to the index. During a schema change backfill, the execution layer's updater must unconditionally write complete index entries to temporary (mutating) indexes for any concurrent update, even if the index's columns are unchanged. This ensures the backfill merger has a complete snapshot to correctly reconcile the final index. If columns are pruned (Rule 1) or writes are simplified away (Rule 2), the temporary index receives incomplete entries (NULLs) or misses the update entirely. Furthermore, missing columns can lead to phantom rows if the partial index predicate evaluates to TRUE when given a NULL value (e.g., WHERE val IS NULL). This change ensures the optimizer always fetches the required columns and avoids simplifying predicate evaluation if the index is a mutation index, correctly propagating the full row state to the execution layer. Fixes: cockroachdb#166122 Release note (bug fix): Fixed a bug where concurrent updates to a table using multiple column families during a partial index creation could result in data loss, incorrect NULL values, or validation failures in the resulting index.
|
Thanks for opening a backport. Before merging, please confirm that it falls into one of the following categories (select one):
Add a brief release justification to the PR description explaining your selection. Also, confirm that the change does not break backward compatibility and complies with all aspects of the backport policy. All backports must be reviewed by the TL and EM for the owning area. |
blathers-crl
bot
added
backport
Member
|
This change is |
|
Detected infrastructure failure (matched: self-hosted runner lost communication with the server). Automatically rerunning failed jobs. (run link) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport 1/1 commits from #168366 (original: #166123).
Release justification: addresses a bug that can lead to partial indexes that are corrupt or fail to construct in the face of concurrent updates.