Skip to content

scheduler: exactly-once dispatch under partition needs consensus (split from #46) #183

Description

@polaris-is-online

Split from #46.

#46 closed on leaderless rendezvous dispatch (HRW over the live host set, commit 75eaaec) plus firing-key idempotency (commits a43c70d, 2cf8964). Together those avoid double dispatch during normal operation and converge the persisted effects when a double dispatch does happen. They do not guarantee exactly-once execution under a network partition.

The residual

When two hosts' views of the live hosts set diverge -- a real partition, or clock skew wide enough that the staleness window disagrees -- each host can compute itself as the rendezvous winner for the same firing and run the full agent loop. The firing-key idempotency keys (firing:<task_id>:<next_run_at>) converge the scheduler rows and the per-firing synced artifacts, so persisted state does not double. But the agent loop itself -- the inference cost, and any non-idempotent side effect it triggers (an external API call, a notification, a tool with real-world effect) -- runs twice.

Why this is its own issue

Eliminating this is exactly-once execution across a partition, which under CAP is a consensus problem. The honest options are the strongly-consistent ones #46 deferred: a Raft-backed lease/coordination service, or quorum-gated dispatch. All of them reintroduce a coordination dependency the leaderless design was specifically avoiding, so this should not be picked up until the cost is actually biting. Today it is bounded: idempotency covers the persisted side, and the only uncovered surface is duplicate inference plus non-idempotent tool effects under a true partition.

Scope

Design-and-spec, not an immediate fix. Tracking so the consensus question has its own home instead of being implied by #46's title.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions