[Security] LeaseMajority split-brain at network-latency boundary

[pathosDev]

Severity / Size

• Severity: HIGH — under specific (rare but realistic) network conditions, two partitions of a cluster can both "win" the split-brain arbitration and both stay up. Classic split-brain symptoms: divergent state, double-write to shared resources, inconsistent shard ownership.
• Size: M (~2d).
• Threat model: not an attacker. This is a timing-and-failure-mode issue: slow network to the lease backend, retry semantics in the underlying Lease implementation, and the framework's defense-in-depth timeout. The split-brain happens when those three layers don't agree on what "acquired" means.

Affected files

• src/cluster/downing/LeaseMajority.ts:120-138 — the acquiring-flag + acquireDeadline recovery path. Comment at line 122-125 says "let the next tick try again" — but if the original acquire eventually succeeded on the backend, this is the door to the bug.
• src/cluster/downing/LeaseMajority.ts:143-158 — runAcquire() writes the cached decision after acquire() resolves. No fencing token; no per-attempt epoch.
• src/coordination/Lease.ts (interface) — defines acquire(): Promise<boolean>. No fencing/epoch concept exposed today.
• src/coordination/KubernetesLease.ts — the K8s-backed implementation. Its retry/timeout semantics are configurable; misalignment with LeaseMajority's 5s default is one root cause.

Background

LeaseMajority is the strongest split-brain resolver — it delegates the equal-size-partition tiebreak to an external arbiter (a Kubernetes Lease object, typically) so both sides of a partition consult the same authority and only one wins. Algorithm is in the file header: strict majority → standard math; strict minority → down self; equal → ask the lease.

The current implementation has a known robustness gap, marked with the inline comment:

if (Date.now() > this.acquireDeadline) {
  this.acquiring = false;       // ← here
}
return new Set();

After the deadline elapses, we abandon the in-flight acquire() and allow the next decide-tick to start a fresh one. The abandoned promise still resolves eventually — and updates this.decision via line 157. But the next tick has already started a new runAcquire. Race window: both can write this.decision, and both could write surviveSet if the lease backend granted both attempts (which can happen if the first attempt's "did I acquire?" response was just slow).

The K8s Lease API actually does serialize acquires per resource — so in practice the first-attempt grant is what wins. But the framework's view of "did I win" is decoupled from the backend's view, because we abandon the first promise. If acquire #1 succeeded on the backend and resolved late, and acquire #2 then took over and the K8s API said "you already hold it" → both attempts return won: true to the framework. Both partitions can do this simultaneously → both think they own the lease → split-brain.

Exploit walkthrough

Setup: 6-node cluster across two AZ's, 3 nodes per AZ. Network partition cuts the inter-AZ link cleanly. Each AZ sees the other as unreachable.

LeaseMajority on both sides falls through to the equal-size branch (3 vs 3). Both kick off lease.acquire().

Step 1 — AZ-1 acquires first: the K8s API processes AZ-1's request at T+0. AZ-1 holds the lease (server-side state). Response packet is generated, sits in the network buffer. Network spike: 6s before the response gets to AZ-1.

Step 2 — acquireTimeoutMs (default 5s) elapses on AZ-1: LeaseMajority.decide() sees Date.now() > acquireDeadline, sets acquiring = false, returns no decision. Next tick will start a fresh runAcquire.

Step 3 — Response arrives at AZ-1 (T+6s): the original runAcquire's promise resolves with won: true. Line 157 writes decision = surviveSet. AZ-1 commits to "we win, down AZ-2". Begins downing AZ-2.

Step 4 — AZ-2 retries acquire: at T+5s (or so) AZ-2's first attempt also timed out (its 5s budget elapsed because the slow link affects both directions). AZ-2's decide() tick fires fresh runAcquire. The K8s API at T+10s tells AZ-2 "you don't hold the lease — AZ-1 does" → AZ-2 gets won: false → decision = downSelfSet → AZ-2 begins downing itself.

OK — that scenario is actually safe. AZ-1 wins, AZ-2 loses. The bug surfaces in a slightly different timing:

Step 4' (the actual race): AZ-2's first acquire times out at T+5s but the response was already sent by the K8s API at T+4.9s saying "you don't hold it" (because AZ-1 already does). Response arrives at AZ-2 at T+10s. In the meantime, at T+8s the K8s lease record's TTL expired (lease.acquire generally takes a TTL). AZ-2's second runAcquire at T+8.5s asks "can I acquire?" — the K8s API says "yes, it's free now" because the TTL elapsed. AZ-2 wins on the second attempt. Then AZ-2's first attempt's slow response arrives — won: false — overwrites the cached decision to downSelfSet. Now AZ-2's state says "we should down ourselves" but AZ-2 already promoted to leader of its partition. And AZ-1's first attempt also resolved late saying "won: true".

Both AZ-1 and AZ-2 have a tick where decision = surviveSet. Both commit to downing the other. Split-brain achieved.

The race is narrow but real, especially when the network has high latency variance or when the user has tuned acquireTimeoutMs aggressively.

How the 8 already-landed security fixes inform this

• Hello-handshake hijack (9c3b005): first-conn-wins on the transport, with explicit rejection of overwrites. Pattern: never let a stale operation overwrite the result of a fresh one. Here we need: never let an abandoned acquire's late response overwrite a fresh acquire's decision.
• FrameDecoder size cap (d454079): patterned "validate at entry, reject early". Here: enforce a hard abort on the abandoned promise — not just an abandoning flag. Cancel the operation if the underlying API supports it.

Fix design

Three coordinated changes.

Change 1 — epoch-stamped attempts.

Each runAcquire() gets a monotonic epoch. When the abandoned promise resolves late, it checks whether its epoch is still the current one; if not, it discards the result silently:

private acquireEpoch = 0;

private async runAcquire(epoch: number, surviveSet, downSelfSet): Promise<void> {
  let won: boolean;
  try {
    won = await this.settings.lease.acquire();
  } catch {
    if (this.acquireEpoch === epoch) this.acquiring = false;
    return;
  }
  if (this.acquireEpoch !== epoch) {
    // We were abandoned — a newer attempt has taken over.
    // If we won server-side, we should release.  See Change 2.
    if (won) void this.settings.lease.release();
    return;
  }
  this.acquiring = false;
  this.decision = won ? surviveSet : downSelfSet;
}

decide(view) {
  // ... existing logic ...
  if (this.acquiring && Date.now() > this.acquireDeadline) {
    this.acquireEpoch += 1;   // invalidate the in-flight attempt
    this.acquiring = false;
  }
  // ...
  this.acquireEpoch += 1;
  this.acquiring = true;
  this.acquireDeadline = Date.now() + (this.settings.acquireTimeoutMs ?? 5_000);
  void this.runAcquire(this.acquireEpoch, surviveSet, downSelfSet);
  return new Set();
}

Late-resolving abandoned acquires can't corrupt the decision anymore.

Change 2 — release on abandon.

If the abandoned promise eventually resolves with won: true, the framework holds a lease it no longer believes it holds. Release it explicitly so the next tick's acquire (or the other partition) can claim it cleanly. Snippet inside runAcquire above.

This requires Lease.release() to be safe-to-call even when ownership is ambiguous. Check KubernetesLease semantics; if the K8s release requires the holder identity to match, the abandoned attempt's release will be a no-op — which is fine.

Change 3 — fencing token (defense-in-depth).

Expose a per-acquire fencing token in the Lease interface:

export interface Lease {
  acquire(): Promise<{ acquired: false } | { acquired: true; token: string }>;
  release(token: string): Promise<void>;
  // Optional: heartbeat(token) for long-held leases.
}

KubernetesLease returns the resource's metadata.resourceVersion as the token. LeaseMajority stores the token alongside the decision. Any subsequent operation that depends on "we own the lease" can re-verify by checking the token is still current (via lease.verify(token)).

This is the strongest defense — but it's an interface change. Optional for the v1 fix; can land in a follow-up if Changes 1+2 are not sufficient.

API surface

// src/coordination/Lease.ts — interface change
export interface Lease {
  acquire(): Promise<boolean>;       // unchanged for v1
  release(): Promise<void>;          // unchanged
  // Future: typed-token return + verify.
}

// src/cluster/downing/LeaseMajority.ts — no public-API change.
//   Internal: acquireEpoch counter; release-on-abandon path.

Backward compatibility

No public-API change in the v1 fix. Existing KubernetesLease / InMemoryLease implementations continue to work — release() is already in the interface. The fencing-token Change 3 is an interface evolution that can land separately.

Test plan

The exploit is a timing race. Tests use a simulated Lease with controllable response latency.

1. Exploit test (tests/unit/cluster/downing/lease-majority-race.test.ts):

   • SimulatedLease where acquire() resolves with won: true after 8s, but the test scheduler advances to 5s, then to 10s.
   • Pre-fix: the abandoned promise overwrites decision; assertion that both partitions can reach decision = surviveSet fails (state-confusion verified).
   • Post-fix: abandoned promise is ignored; decision reflects only the current-epoch attempt.

2. Defense test: epoch-bump scenarios.

   • Attempt 1 acquires (slow); attempt 2 starts with new epoch; attempt 1 resolves → silently discarded; attempt 2's result is the active decision.

3. Release-on-abandon test: assert Lease.release() is called when an abandoned acquire resolves won: true.

4. Regression: existing LeaseMajority.test.ts tests pass without modification.

5. Multi-node integration: 4-node cluster + InMemoryLease; partition into 2v2; both sides start acquire; one wins; other downs self. Verify symmetric outcome.

Acceptance criteria

• LeaseMajority.acquireEpoch counter implemented.
• Abandoned acquires release lease + don't overwrite decision.
• Exploit test demonstrates the pre-fix race + post-fix block.
• Defense tests cover the epoch-bump + release-on-abandon paths.
• All existing downing tests still pass.
• (Optional) Fencing-token Change 3 designed as a follow-up issue if the v1 fix is insufficient under load.
• Plan-doc + README "Known security caveats" entry updated on land.