Skip to content

History / shard election

Revisions

  • docs: say that the election is in review, not shipped The page claimed "shipped, flag-off" while both PRs were still open. Writing a page from the diff and dating it as landed is the same merged-is-not-deployed conflation this repo keeps getting caught by, one artefact earlier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

    @akuksin akuksin committed Oct 11, 2026
  • docs: the shard election, and the three env vars that arm it Covers noetl/worker's C2: the per-shard election over coordination.k8s.io/v1 Leases that mints the token NOETL_EHDB_FENCING enforces against. Shipped flag-off; the RBAC it needs does not exist on prod yet. Rule 2a: the deployment-spec page is the env-var source of truth, and the code change adds three — NOETL_EHDB_ELECTION, NOETL_EHDB_ELECTION_LEASE_SECS, NOETL_EHDB_ELECTION_RENEW_SECS. The first was missing entirely even though NOETL_EHDB_FENCING, which is useless without it, was already documented. New page records what a reader cannot recover from the diff: * why the unit of exclusion is the pod ordinal and not the bus (one writer hosts two L0 engines on the same PVCs); * ⚠⚠ why the holder identity carries a per-process uuid — a StatefulSet pod name is stable, so a name-only identity lets a RESTARTED writer renew its predecessor's lease and silently inherit its epoch, which is the split-brain the epoch exists to catch; * why the driver is fail-soft, and that this is load-bearing rather than defensive: the prerequisite Role binding does not exist, so a premature flip produces 403 on every tick, and exiting over that would crash-loop the one StatefulSet whose restart behaviour is the riskiest part of the change; * ⚠⚠ that a fence must consult ehdb_election_token_valid and NOT holds_lease — the former expires on the local clock, which is the only way the claim can be made while partitioned; * why the driver is an OS thread (KubeLeaseStore block_on's its own runtime; tokio::spawn would panic on the first tick). Refs noetl/ai-meta#460 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

    @akuksin akuksin committed Oct 11, 2026