Allow setting dictionary compression per cluster and replica#37700
Conversation
def-
left a comment
def-
left a comment
There was a problem hiding this comment.
I'd like to see random dictionary compressions being set per cluster/replica in parallel-workload. Maybe in the existing FlipFlagsAction, which has to be extended a bit for that. Tell me if you want me to implement that instead
DAlperin
left a comment
DAlperin
left a comment
There was a problem hiding this comment.
Approve mod one question. Also, not localized to this PR but man, this is a lot of boilerplate to add one field
| EXPERIMENTAL => { | ||
| self.expect_keywords(&[ARRANGEMENT, COMPRESSION])?; | ||
| ReplicaOptionName::ExperimentalArrangementCompression | ||
| } |
There was a problem hiding this comment.
Are we sticking with the EXPERIMENTAL prefix or are we just releasing?
There was a problem hiding this comment.
Yeah, we'll leave it as such for the moment and eventually accept it without EXPERIMENTAL, if we decide to go with it.
|
If you're up for it I'd appreciate your input to randomize the flag! |
df5a143 to
e62e5fa
Compare
|
I don't have permission to push to your branch, so pushed to mine instead: https://github.com/def-/materialize/tree/pr-test7 |
Turn arrangement dictionary compression from a purely global feature flag into a per-cluster and per-replica configuration modeled after the introspection options. Add the `EXPERIMENTAL ARRANGEMENT COMPRESSION` option to `CREATE CLUSTER`, `CREATE CLUSTER REPLICA`, and `ALTER CLUSTER`, store it durably in the catalog (v90), and thread it through the cluster controller shape model so a change reprovisions controller-owned replicas. The `enable_arrangement_dictionary_compression_alpha` flag now only gates the feature at replica-creation time: a replica honors its configured value only while the flag is enabled, so the configuration can be set and stored regardless of the flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add the objects_v90 proto serialization snapshot required by test_proto_serialization_stability. * Regenerate the test_persist_open initial-snapshot insta golden for the new arrangement_compression field. * Version-gate the SHOW CREATE CLUSTER platform-check goldens: the new EXPERIMENTAL ARRANGEMENT COMPRESSION option only appears at v2603400. * Make the arrangement-dictionary-compression regression test request compression per cluster, since the gating flag alone no longer enables it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
306ff05 to
ea362c7
Compare
ALTER CLUSTER SET SIZE on a cluster with replicas is not reflected in SHOW CREATE CLUSTER without zero-downtime reconfiguration, which is gated behind a private-preview flag. Match the golden to actual output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for the reviews! |
Follow-up to #37700, which merged after main moved to v26.35.0-dev.0 Seen in https://buildkite.com/materialize/nightly/builds/17499 Test run: https://buildkite.com/materialize/nightly/builds/17501
3 participants
Motivation
Closes CPU-173
Arrangement dictionary compression was a purely global feature flag captured at replica creation. This makes it configurable per cluster and per replica, modeled after the introspection options.
Description
Adds the
EXPERIMENTAL ARRANGEMENT COMPRESSIONoption toCREATE CLUSTER,CREATE CLUSTER REPLICA, andALTER CLUSTER SET/RESET, stored durably in the catalog (version bump to v90). The value is threaded through the cluster-controller shape model, so changing it reprovisions controller-owned replicas like a size or introspection change does.enable_arrangement_dictionary_compression_alphanow only gates the feature: a replica honors its configured value only while the flag is enabled, and the gate is evaluated at replica-creation time. The configuration is always accepted and stored regardless of the flag, so a cluster can be pre-configured and start honoring it once the flag is enabled on new or restarted replicas.