bundle full schema and constraint catalogs in partition snapshots#61
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #57 and #58. After PR #56 surfaced "constraints don't reach all nodes uniformly" as a follow-up, this PR fixes the late-join half of that gap.
Schemas and constraints already broadcast to every alive node on write (
replication_ops.rs:81-85and:345-369— likely added in #54 after the issues were filed). The remaining hole is the snapshot path:SchemaStore::export_for_partitionandConstraintStore::export_for_partitionfilterentry.partition() == partition, so a joining node only ever received entries whoseschema_partition(entity)happened to hash into a partition it owned. Catalog entries under any other partition were unreachable through snapshots — a node could rebalance into the cluster missing most of the catalog, and a freshly-joined 4th node empirically held 0 constraints in the #57 trace.Both stores now expose
export_all(), which serializes every entry regardless of partition using the same wire format asexport_for_partition(soimport_schemas/import_constraintsare unchanged).StoreManager::export_partitionuses these forDB_SCHEMAandDB_CONSTRAINT. Catalogs are small in-memory state and import is idempotent overwrite, so re-applying on every partition snapshot a joining node fetches is safe.A shared private
serialize_entrieshelper in each store removes the duplicated count-prefixed loop between the per-partition and full-catalog exports.Test plan
cargo test -p mqdb-cluster --lib— 3 new tests pass:schema_store::export_all_carries_every_schema_regardless_of_partitionconstraint_store::export_all_carries_every_constraint_regardless_of_partitionpartition_io::partition_snapshot_carries_full_schema_and_constraint_catalog— multi-entity setup spanning many partitions; asserts dst has the full catalog after a single-partition snapshot, plus full counts match source.partition_io::export_import_roundtrip_covers_all_db_storesupdated — previously asserted per-partition isolation for schemas/constraints, now asserts they arrive regardless of partition.cargo make clippy— clean (pedantic, all targets + wasm).cargo make format-check— clean.cargo make test— full workspace clean.examples/cluster-rebalance-stores/run.shactually run twice with an enterprise license generated locally:schema get posts/comments via node 4, all duplicates rejected, FK orphan rejected); FK cascade removed 17 of 19 eligible children. The single hard-assertion failure ("18 of 20 extra child comments") is a confirmed-flaky cluster-warmup transient unrelated to schema/constraint replication.Out of scope
Remaining FK-cascade survivors that still show up under hash-unlucky distributions are #22 (cluster cascade does not filter by ownership on remote nodes) — separate follow-up.