Split from #46.
#46 closed on leaderless rendezvous dispatch (HRW over the live host set, commit 75eaaec) plus firing-key idempotency (commits a43c70d, 2cf8964). Together those avoid double dispatch during normal operation and converge the persisted effects when a double dispatch does happen. They do not guarantee exactly-once execution under a network partition.
The residual
When two hosts' views of the live hosts set diverge -- a real partition, or clock skew wide enough that the staleness window disagrees -- each host can compute itself as the rendezvous winner for the same firing and run the full agent loop. The firing-key idempotency keys (firing:<task_id>:<next_run_at>) converge the scheduler rows and the per-firing synced artifacts, so persisted state does not double. But the agent loop itself -- the inference cost, and any non-idempotent side effect it triggers (an external API call, a notification, a tool with real-world effect) -- runs twice.
Why this is its own issue
Eliminating this is exactly-once execution across a partition, which under CAP is a consensus problem. The honest options are the strongly-consistent ones #46 deferred: a Raft-backed lease/coordination service, or quorum-gated dispatch. All of them reintroduce a coordination dependency the leaderless design was specifically avoiding, so this should not be picked up until the cost is actually biting. Today it is bounded: idempotency covers the persisted side, and the only uncovered surface is duplicate inference plus non-idempotent tool effects under a true partition.
Scope
Design-and-spec, not an immediate fix. Tracking so the consensus question has its own home instead of being implied by #46's title.
Split from #46.
#46 closed on leaderless rendezvous dispatch (HRW over the live host set, commit 75eaaec) plus firing-key idempotency (commits a43c70d, 2cf8964). Together those avoid double dispatch during normal operation and converge the persisted effects when a double dispatch does happen. They do not guarantee exactly-once execution under a network partition.
The residual
When two hosts' views of the live
hostsset diverge -- a real partition, or clock skew wide enough that the staleness window disagrees -- each host can compute itself as the rendezvous winner for the same firing and run the full agent loop. The firing-key idempotency keys (firing:<task_id>:<next_run_at>) converge the scheduler rows and the per-firing synced artifacts, so persisted state does not double. But the agent loop itself -- the inference cost, and any non-idempotent side effect it triggers (an external API call, a notification, a tool with real-world effect) -- runs twice.Why this is its own issue
Eliminating this is exactly-once execution across a partition, which under CAP is a consensus problem. The honest options are the strongly-consistent ones #46 deferred: a Raft-backed lease/coordination service, or quorum-gated dispatch. All of them reintroduce a coordination dependency the leaderless design was specifically avoiding, so this should not be picked up until the cost is actually biting. Today it is bounded: idempotency covers the persisted side, and the only uncovered surface is duplicate inference plus non-idempotent tool effects under a true partition.
Scope
Design-and-spec, not an immediate fix. Tracking so the consensus question has its own home instead of being implied by #46's title.