Skip to content

adapter: move a replacement's expression cache entries to its target on apply - #38682

Draft
ggevay wants to merge 2 commits into
MaterializeInc:mainfrom
ggevay:gabor/sql-683-expr-cache-restamp-on-apply
Draft

adapter: move a replacement's expression cache entries to its target on apply#38682
ggevay wants to merge 2 commits into
MaterializeInc:mainfrom
ggevay:gabor/sql-683-expr-cache-restamp-on-apply

Conversation

@ggevay

@ggevay ggevay commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

This is on top of #38677, so please review the "adapter: record the item version in expression cache entries" commit there.

Motivation

#38677 fixes SQL-683 by having ExpressionCache::open drop entries recorded for an earlier definition of an item. After an ALTER MATERIALIZED VIEW ... APPLY REPLACEMENT that leaves the next boot re-optimizing the replaced materialized view once, and nothing in the logs says why an entry was dropped. This PR is stacked on #38677 (its first commit); review the second commit.

Description

After a successful apply, the coordinator asks the cache to move the replacement's own entries to the surviving item: the local entry moves under the item's root id, the global entry stays under the writing id, both are recorded at the new version, and the item's other entries are removed. The moved entries keep the optimizer features that produced them, so a feature change between the replacement's creation and the apply makes the next boot re-optimize the view like any other item. Correctness does not rest on this write, the version check in #38677 does that; it only spares the boot after an apply one re-optimization per applied replacement.

The move runs inside the cache task (ExpressionCache::restamp), reading the replacement's entries from the durable cache's local mirror. An entry that is unreadable, or missing from this process's view of the cache, is skipped and removed, leaving the next open to re-optimize what it would have served. The read and the rewrite are not atomic against another process of the same build version, which is safe because ids are never reused: whatever such a process writes under the replacement's id describes the definition being adopted.

ExpressionCache::open now logs how many entries it kept and how many it removed, by reason (dropped item, item version mismatch, dropped index import, unreadable, prior build version), so a cache miss can be attributed after the fact. An entry the running build cannot read is dropped instead of being reported on every boot; entries are only ever read by the build version that wrote them, so this can only affect entries the same build wrote.

Tests

  • expr_cache unit test: the moved entries of an apply are returned under the target's ids at the new version, and the entries the move invalidates are gone even where the version check would have kept them.
  • test_replacement_materialized_view_expression_cache_hit_after_apply (environmentd server tests): creates an index on the replacement's input after the replacement was planned, applies, restarts, and checks through mz_compute_dependencies that the view's dataflow still reads the input directly. A re-optimization at boot adopts the index, which is what happens without this change.

Alternatives

Writing the surviving item's in-memory plans to the cache after the apply, stamped with the cluster's current optimizer features, would avoid the re-optimization without a new cache operation. But the apply moves the replacement's plans over without re-optimizing, so a feature change between the replacement's creation and the apply would stamp an old-feature plan with the new features, and every later boot would hit and reinstall it while re-optimizing everything else on the cluster. Moving the replacement's own entries keeps the features that produced each plan.

Part of SQL-683.


🤖 Generated with Claude Code

ggevay and others added 2 commits September 5, 2026 20:48
Applying a materialized view replacement changes an item's definition
while the item keeps its GlobalIds. The invalidation on apply only
reaches cache entries under the applying process's build version, so a
0dt deployment that cached the pre-apply expressions under its own build
version during its read-only phase reads them again after promotion:
bootstrap panics with "catalog out of sync" if an old dependency was
dropped, and otherwise the old query can be rendered and served.

Record the latest RelationVersion of the owning item with each entry and
drop entries recorded at another version when the cache is opened. The
apply bumps the target's version, so stale entries are dropped whoever
wrote them.

Tests: a unit test for the version check, a server test that reproduces
the bootstrap panic without the fix, and a platform check that applies a
replacement and drops its old input in the first manipulate phase, which
the zero-downtime upgrade scenarios run inside the rollout window.

Closes: SQL-683

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…on apply

After a successful apply, the cache moves the replacement's own entries
to the surviving item's ids at its new version and removes the item's
other entries, so the next boot hits instead of re-optimizing the view.
The moved entries keep the optimizer features that produced them, so a
feature change between the replacement's creation and the apply makes
the next boot re-optimize the view like any other item. Nothing depends
on the write landing: entries recorded before the apply carry the old
version and the next open drops them either way.

The cache open now logs how many entries it kept and removed, by reason,
and drops entries the running build cannot read instead of reporting
them on every boot.

Tests: a unit test for the move, and a server test that creates an index
on the replacement's input after the replacement was planned and checks
that the boot after the apply still reads the input directly, which a
re-optimization would not.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@ggevay
ggevay force-pushed the gabor/sql-683-expr-cache-restamp-on-apply branch from 66ea566 to 8f64fc9 Compare September 5, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant