shard election

Shard election (C2) — minting the fencing token

The per-shard leader election the writer runs over Kubernetes coordination.k8s.io/v1 Leases. It is C2 of noetl/ai-meta#460; the fence that acts on its output is C3 and is a separate decision.

Status: open in review, flag-off — the store in noetl/worker#352, the driver in #353. ⚠ Neither is merged, so this page describes code that is not yet in any release; "merged" and "running in prod" are separate questions again after that. NOETL_EHDB_ELECTION defaults to false, nothing in prod sets it, and the RBAC it needs does not exist yet.

What it is for

NOETL_EHDB_FENCING=enforce rejects writes from a superseded writer by comparing fencing tokens. Until this landed there was no token: ShardElection (noetl/ehdb#331) was implemented, tested and merged with no call site anywhere, so every writer's epoch was 0 and /metrics published a hard-coded 0.

⚠⚠ The order is non-negotiable. A fence enforces against an epoch, so arming enforce while nothing mints does not degrade — it fails every writer at once. Election live and minting first, verified on the scrape; enforce after.

Single-writer-per-shard currently rests on StatefulSet replicas: 1, which is an orchestration preference, not a mutual-exclusion primitive.

Where it runs, and at what granularity

Spawned from the writer's boot path (Worker::new) only where this process hosts a writer — not on worker pods, which merely claim from a shard. A worker electing itself for a shard it does not write would mint tokens nobody can act on.

⚠ One election per process, not per bus. A writer pod hosts two L0 engines (command bus and event bus) and the entrypoint derives both shards from the same StatefulSet ordinal. Both are backed by the same pod's PVCs, so the thing being excluded is two processes owning one ordinal's volumes — a per-bus lease would be two names for one problem. The lease is ehdb-shard-<shard:08x>.

⚠⚠ Why the holder identity carries a per-process suffix

The identity is <hostname>#<uuid>, minted once per process.

A StatefulSet pod name is stable — the writer is always noetl-cmdbus-writer-0. An identity of just the pod name makes a restarted writer look like the same holder to its predecessor's lease: renew() succeeds and the new process silently inherits the old epoch. A fence could then not distinguish the restarted writer from a zombie predecessor, which is exactly the split-brain the epoch exists to catch. This is pinned by a_restarted_writer_inherits_the_epoch_unless_identity_is_per_process, which asserts the inheriting behaviour for a name-only identity and the correct behaviour with the suffix — a positive control for the decision, not just a test of the happy path.

The cost is explicit, and it is the failover window: a restarted writer cannot acquire until the previous lease expires, so it is unelected for up to NOETL_EHDB_ELECTION_LEASE_SECS (default 15 s).

⚠ The boundary is a strict >: at exactly renewed_at + duration the lease is still live. Asserted, because it is the number a failover bound is computed from.

⚠ Fail-soft: wrong RBAC degrades, it does not crash-loop

Every lease-store error is counted by reason, logged (first occurrence, then every 60th), and otherwise ignored. The loop never exits and never panics, and a flag-on election in an environment that cannot support it logs and declines rather than failing Worker::new.

This is load-bearing rather than defensive. The prerequisite Role binding does not exist yet, so the first thing a premature flip produces is 403 Forbidden on every tick — and a writer that exited over that would crash-loop the one StatefulSet whose restart behaviour is the riskiest part of the change.

What is not fail-soft is the claim to hold a token. See below.

⚠⚠ token_valid is the series a fence must consult, not holds_lease

series	means
ehdb_election_active	a driver is running in this process. Not that a token exists
ehdb_election_epoch	the last token this writer held. 0 = never issued
ehdb_election_holds_lease	the last successful call left us as holder
ehdb_election_token_valid	holder and last contact fresher than the lease duration
ehdb_election_last_contact_age_seconds	seconds since the last successful call; -1 = never
ehdb_election_lease_duration_seconds / ..._renew_interval_seconds	the configured pair, after clamping
ehdb_election_outcomes_total{outcome}	acquired / renewed / lost / held_by_other
ehdb_election_errors_total{reason}	forbidden / unauthorized / unreachable / other

holds_lease is what the last successful call said. token_valid additionally requires that the call was recent enough that the lease cannot have lapsed since — decided locally, with no API-server call, which is the only way the claim can be made while partitioned. A writer cut off from the API server keeps holds_lease=1 and drops to token_valid=0 once contact is older than the lease could survive; a lease we cannot renew is one another node is entitled to take.

A renewal deliberately does not advance the epoch. Minting a token per renewal would make the store fence the holder against itself.

⚠ Every series renders in both states — election running and not — and every label value is pinned at 0. A labelled metric is absent from a scrape until something increments it, so an unpinned reason="forbidden" would be missing both when RBAC is correct and when the build predates the label, two states an operator must be able to tell apart. -1 for "never contacted" is the same discipline: 0 would claim contact just now.

The RBAC it needs (owner actions, not done)

1. A dedicated ServiceAccount for the cmdbus-writer. It currently runs as default — the only workload in noetl that does.
2. serviceAccountName set on sts/noetl-cmdbus-writer. ⚠ A pod-template change, so it rolls the writer.
3. The ehdb-shard-election Role bound to that SA: leases get,list,watch,create,update, no delete.

⚠ Binding to default instead would grant lease create/update to every pod in the namespace that defaults its SA — for a Role whose whole purpose is mutual exclusion of writers, that is the wrong shape of mistake.

⚠ default carries no iam.gke.io/gcp-service-account annotation and the writer reads no GCP credentials, so a new SA without an annotation is GCP-identity equivalent. Reusing an existing named SA (noetl-worker, noetl-server-rust) would instead hand the writer a cloud identity it does not need.

⚠ Why the driver is an OS thread, not a tokio task

KubeLeaseStore bridges the sync LeaseStore trait onto reqwest by calling block_on on a runtime of its own, and tokio panics when block_on runs on a thread already driving tasks. A plain tokio::spawn would panic on the first tick — in the writer, in prod, behind a flag that had looked harmless. The dedicated thread also means a stalled API server cannot stall the writer's reactor: the worst case is a stale epoch, not back-pressure on dispatch.

Related

• Deployment specification — the env-var catalogue.
• cmdbus writer / tier service
• noetl/ehdb#331 — ShardElection.
• ehdb/docs/spec/lease-election-k8s-binding.md — the k8s field mapping, and the ⚠ gap it did not cover: PUT on an absent Lease returns 201 Created and creates it, even with resourceVersion set.