release-26.2: sql/catkv: refresh stale name->ID mappings in SystemDatabaseCache#169139
Merged
Merged
Conversation
The per-node SystemDatabaseCache holds name->ID mappings for system
database namespace entries for the lifetime of the process. Until now it
treated the first observation of any name as authoritative: once a name
was cached, subsequent observations of the same name with a different ID
were silently dropped. The comment on the type even codified this as an
assumption ("for a given name and a given cluster version, the name to
ID mapping will never change for the life of process").
That assumption does not hold for PCR reader tenants. A reader tenant is
bootstrapped with its own dynamically-allocated IDs for system tables
that lack a fixed reserved-range ID (e.g. system.privileges), and then
SetupOrAdvanceStandbyReaderCatalog rewrites the namespace to point those
names at the externalized descriptors replicated from the source tenant,
which carry the source's IDs.
Any cache entry populated between reader tenant startup and that
rewrite -- and the very first user authentication populates one for
system.privileges via the synthetic privilege cache -- pins the
bootstrap-time ID forever. Since the rewrite also deletes the descriptor
at the bootstrap-time ID, every subsequent name resolution that hits the
cache returns an ID that no descriptor exists for, and lookups fail with
"resolved <name> to <id> but found no descriptor with id <id>".
Restarting the reader tenant masks the bug because it discards the
cache.
This change permits the cache to replace an existing entry when an
incoming observation has the same name but a different ID and a strictly
newer MVCC timestamp than the cached entry. The MVCC timestamp guard
ensures a stale read at an older AOST cannot regress a fresher mapping,
and the same check is repeated under the write lock in update so a
concurrent fresher write cannot be clobbered by a slower stale one.
The warm-up entries inserted at construction carry a zero MVCC
timestamp, so any real KV-sourced observation dominates them, which is
the desired behavior.
Resolves cockroachdb#153555.
Epic: none
Release note (bug fix): A physical cluster replication reader tenant no
longer fails authentication and other queries with errors of the form
"resolved <name> to <id> but found no descriptor with id <id>" after the
reader tenant ingests a system table at an ID different from the one it
was bootstrapped with. Previously, a per-node namespace cache could pin
the bootstrap-time ID and require a tenant restart to recover.
Reverts 767fcb6 ("roachtest: delake pcr roachtests that offset reader tenant system tables") and 20ea8e1 ("roachest: ensure reader tenant accepts connections before restart"), which together restarted the reader tenant in PCR roachtests after the standby read poller had advanced. The restart was a workaround for the SystemDatabaseCache bug fixed in the previous commit: a stale name->ID mapping for system.privileges would survive in the per-node cache and cause queries to fail with "resolved privileges to <id> but found no descriptor". Restarting the tenant discarded the cache. With the cache itself refreshed correctly, the workaround is no longer needed and the test goes back to exercising the same flow a customer would see. Epic: none Release note: none
The doctype comment still claimed that the name->ID mapping never changes for the life of the process. The previous commit relaxed that to permit replacement when an incoming observation carries a different ID at a strictly newer MVCC timestamp. Update the comment to match. Epic: none Release note: None
|
Thanks for opening a backport. Before merging, please confirm that the change does not break backwards compatibility and otherwise complies with the backport policy. Include a brief release justification in the PR description explaining why the backport is appropriate. All backports must be reviewed by the TL for the owning area. While the stricter LTS policy does not yet apply, please exercise judgment and consider gating non-critical changes behind a disabled-by-default feature flag when appropriate. |
Contributor
|
😎 Merged successfully - details. |
Member
msbutler
approved these changes
Apr 27, 2026
5780c2b
into
cockroachdb:release-26.2
33 of 34 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport 3/3 commits from #169044.
/cc @cockroachdb/release
The per-node
SystemDatabaseCacheholds name→ID mappings for system database namespace entries for the lifetime of the process. Until now it treated the first observation of any name as authoritative: once a name was cached, subsequent observations of the same name with a different ID were silently dropped. The comment on the type even codified this as an assumption ("for a given name and a given cluster version, the name to ID mapping will never change for the life of process").That assumption does not hold for PCR reader tenants. A reader tenant is bootstrapped with its own dynamically-allocated IDs for system tables that lack a fixed reserved-range ID (e.g.
system.privileges), and thenSetupOrAdvanceStandbyReaderCatalogrewrites the namespace to point those names at the externalized descriptors replicated from the source tenant, which carry the source's IDs.Any cache entry populated between reader tenant startup and that rewrite — and the very first user authentication populates one for
system.privilegesvia the synthetic privilege cache — pins the bootstrap-time ID forever. Since the rewrite also deletes the descriptor at the bootstrap-time ID, every subsequent name resolution that hits the cache returns an ID that no descriptor exists for, and lookups fail withresolved <name> to <id> but found no descriptor with id <id>. Restarting the reader tenant masks the bug because it discards the cache.This change permits the cache to replace an existing entry when an incoming observation has the same name but a different ID and a strictly newer MVCC timestamp than the cached entry. The MVCC timestamp guard ensures a stale read at an older AOST cannot regress a fresher mapping, and the same check is repeated under the write lock in
updateso a concurrent fresher write cannot be clobbered by a slower stale one. The warm-up entries inserted at construction carry a zero MVCC timestamp, so any real KV-sourced observation dominates them, which is the desired behavior.Resolves #153555.
Epic: none
Release note (bug fix): A physical cluster replication reader tenant no longer fails authentication and other queries with errors of the form
resolved <name> to <id> but found no descriptor with id <id>after the reader tenant ingests a system table at an ID different from the one it was bootstrapped with. Previously, a per-node namespace cache could pin the bootstrap-time ID and require a tenant restart to recover.Release justification: bug fix for PCR reader tenant authentication failures