Skip to content

cluster unique: empty-map sync self-seal + bootstrap founding speed-up #106

Description

@fabracht

Follow-up to #101 / PR #105. PR #105 closed the empty-voter fallback (a node with no established unique-voter set no longer seals over the full membership). A related empty-map hole remains open.

The hole

The synchronous seal_or_queue_unique self-seals when unique_majority() == 1. unique_voter_group() currently returns {self} whenever the voter set is empty and unique_fanout_group().len() <= 1 — and an empty/unsynced local partition map also satisfies that (all_nodes() is empty ⇒ fan-out is {self}). So a node right after a restart, before update_partition_map installs the raft map (the event loop calls become_primary before installing the map), treats itself as single-node and self-seals every partition it is named primary of, skipping merge_for_seal — then, once it adopts the voter set, it can serve reserves against a store that is missing reservations it did not receive while down.

Scope: narrow, restart-only oversell. Committed claims are backstopped by the record-driven reconciler; the residual is the uncommitted window. Reproduced by a unit test (empty_map_node_does_not_self_seal in the reverted work): on today's code the node self-seals (unique_sealed == Some(epoch)); it should fail closed (None).

Why the straightforward fix isn't enough

The obvious guard — in the empty-voter branch of unique_voter_group(), self-serve only when all_nodes() == {self} (a genuine single-node cluster), else fail closed — closes the hole and passes the full unit + integration suite and the V7 model. But a real-cluster smoke caught a regression the unit tests and an analytical quorum both missed: removing the fast (buggy) self-seal made a fresh 3-node cluster take ~60–80s to fully accept unique creates (creates fail 503 not-yet-durable, recovering gradually per partition). Root cause: a partition can only seal via the proper path once its primary has a non-empty voter set, and on a fresh cluster the leader founds voters = all_nodes() — which is just {leader} before the rebalance distributes partitions — then paces promotion up to the full set over ~20s, on top of formation epoch churn.

That trade (a narrow restart-only oversell for a ~60–80s degraded-serving window on every bootstrap) is why #4 was reverted from PR #105 rather than shipped.

Proposed fix

  1. Fail-closed guard: distinguish a genuine single-node cluster (all_nodes() == {self}) from an unsynced empty map (all_nodes() == []) in unique_voter_group()'s empty-voter branch; the latter fails closed.
  2. Bootstrap founding speed-up: so the guard doesn't slow bootstrap, safely found voters = full membership at once on a fresh cluster (safe — no reservations exist yet — matching the V5/V6 "founding takes the whole membership at once" rule) instead of founding {leader} and pacing up. Consider deferring the founding until membership/partition assignment has settled, or founding from the raft membership rather than all_nodes().

Acceptance

  • Empty-map node fails closed (unit test, before/after).
  • Real-cluster 3-node bootstrap accepts unique creates within a few seconds (no ~60–80s window).
  • TLA: extend specs/ClusterUniqueReconfigV7.tla (or a V8) to cover the founding/bootstrap transition, not just the abstract "empty ⇒ seal disabled".
  • Real-cluster smoke required (this is exactly the class of regression unit tests + analytical review missed).

Documented in docs/design/cluster-unique-hardening.md under "Open follow-up — empty-MAP sync self-seal".

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions