adapter: don't leak arrangement compression into the realized config - #38112
Draft
aljoscha wants to merge 1 commit into
Draft
adapter: don't leak arrangement compression into the realized config#38112aljoscha wants to merge 1 commit into
aljoscha wants to merge 1 commit into
Conversation
aljoscha
marked this pull request as draft
August 7, 2026 11:49
aljoscha
force-pushed
the
aljoscha/cluster-legacy-06-compression-fix
branch
2 times, most recently
from
August 7, 2026 12:28
6ce5344 to
cf73bc0
Compare
Arrangement compression is a replica config shape dimension: it is part of `ReconfigurationTarget`, the reconcile kernel distinguishes replicas by it, and the controller transitions it through the hydrate-overlap like SIZE. Several hand-maintained dimension lists forgot it. The reshape path writes the reconfiguration record while resetting the config shape carried by `new_config` back to the realized values, deferring the transition to the record's cut-over. The reset enumerated size, replication factor, availability zones, and logging, but missed arrangement compression. A compression-carrying ALTER thus flipped the realized config immediately: the baseline strategy then desired the new shape and bounced the baseline replicas mid-reconfiguration, defeating the graceful overlap, and the cut-over later wrote the same value again. Reset via `realized_reconfiguration_target` and `apply_reconfiguration_target` instead of hand-listing dimensions, so a target dimension can no longer be missed here. The planner's WAIT gate had the same gap: it rejected a `WAIT` clause on a compression-only ALTER as having nothing to wait on, even though such an ALTER transitions through a hydrate-overlap like any other shape change. Allow it. The observability surface had it too: `mz_cluster_reconfigurations.changes` diffed only the other four dimensions, so a compression-only record reported an empty diff while in progress, and the SHOW CLUSTERS activity summary showed such a cluster as steady. Both now report the dimension. The compression ALTERs in managed_cluster.slt relied on the realized-config leak: their SHOW CREATE readback only saw the new value because the reshape wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'), which advances the realized config in the ALTER's own transaction. ALTER ... RESET takes no WITH clause, so the RESET spelling is exercised for acceptance without a readback. cluster-controller.td gains regression coverage: the cc_preserve section asserts that a folding compression ALTER lands in the record's target and changes while the realized config stays untouched and only the overlap replica is bounced, and a new section runs a compression-only ALTER through the record path end to end. Changing the MV's definition changes its builtin fingerprint, so the builtin schema migration registers a replacement step for `mz_cluster_reconfigurations`, and the catalog-docs golden and the mz_show_clusters plan golden are rewritten.
aljoscha
force-pushed
the
aljoscha/cluster-legacy-06-compression-fix
branch
from
August 7, 2026 12:28
cf73bc0 to
8f11dd9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #38104.
Arrangement compression is a replica config shape dimension: it is part of
ReconfigurationTarget, the reconcile kernel distinguishes replicas by it, and the controller transitions it through the hydrate-overlap likeSIZE. Several hand-maintained dimension lists forgot it.The reshape leak.
reshape_alter_cluster_managedwrites the reconfiguration record while resetting the config shape carried bynew_configback to the realized values, deferring the transition to the record's cut-over. The reset enumerated size, replication factor, availability zones, and logging, but missed arrangement compression. A compression-carryingALTERthus flipped the realized config immediately: the baseline strategy then desired the new shape and bounced the baseline replicas mid-reconfiguration, defeating the graceful overlap, and the cut-over later wrote the same value again. The reset now goes throughrealized_reconfiguration_target/apply_reconfiguration_targetinstead of hand-listing dimensions, so a target dimension can no longer be missed at this site.The planner gate. The
WAITgate rejected a compression-onlyALTERas having nothing to wait on, even though such anALTERtransitions through a hydrate-overlap like any other shape change. It is now allowed (and the error message names the option).The observability surface.
mz_cluster_reconfigurations.changesdiffed only the other four dimensions, so a compression-only record reported an empty diff while in progress, and theSHOW CLUSTERSactivity summary showed such a cluster as steady mid-reconfiguration. Both now report the dimension.Tests. The compression
ALTERs inmanaged_cluster.sltrelied on the leak: theirSHOW CREATEreadback only saw the new value because the reshape wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'), which advances the realized config in theALTER's own transaction.ALTER ... RESETtakes noWITHclause, so theRESETspelling is exercised for acceptance without a readback.cluster-controller.tdgains regression coverage: thecc_preservesection asserts that a folding compressionALTERlands in the record's target andchangeswhile the realized config stays untouched and only the overlap replica is bounced, and a new section runs a compression-onlyALTERthrough the record path end to end.Known gap, left for a follow-up: the audit event's reconfiguration details intentionally omit compression, and a shape-dimension
RESETcannot carry aWAITclause (the parser only acceptsWITHonSET).Motivation
Fixes a bug in a feature that has yet to be released.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.