Skip to content

docs: federation design — one hub view over N machines - #317

Merged
saucam merged 3 commits into
mainfrom
docs/federation-design
Sep 5, 2026
Merged

docs: federation design — one hub view over N machines#317
saucam merged 3 commits into
mainfrom
docs/federation-design

Conversation

@saucam

@saucam saucam commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator

What

Adds docs/federation-design.md — a proposal for making the fleet span machines (laptop, workstation, Hetzner, a future Mac mini) under one conductor, plus the local-mode.md / POSITIONING.md edits that keep the three docs consistent.

Design only. No code.

Why

The conductor is designed for one daemon, and the gap is structural rather than a missing feature:

Primitive Machine-aware?
FleetSessionView (fleet.ts:33) no
machine_map() (fleet.ts:314) no — derives workspaces from the local session list
sessions / dispatch_tasks (store.ts:262, :396) no host column
episodes (memory/store.ts:246) workspace_id only
fleet.subscribe (schemas.ts:365) scope: z.literal("tenant")

So "show me every session across my machines" is not merely absent — it is inexpressible, because workspaceId conflates which codebase with where it runs. No open issue covers this (#279, #280, #245, #251 are all single-daemon).

The seams that already exist

Most of the weight is already carried:

  • resolveWorkdirAlias() (share/git-alias.ts:37) already computes the cross-machine join key (github.com/owner/repo), written for session export
  • share/path-rewrite.ts already re-anchors absolute paths
  • dispatch_tasks is already a distributed queue — atomic claim, claim_owner, stale reclaim, backoff, failure-limit — that happens to run in one process
  • schemas.ts:365 comments that widening scope past "tenant" was anticipated; fleet-board.test.ts:121 asserts "machine" is currently rejected

Decisions worth grilling

  • Retrieval: mirror cards, leave episodes, scatter-gather. The corpus stays on the machine that produced it; the hub fuses top-k with RRF. Cost is that p95 is bounded by the slowest satellite, so fan-out carries a deadline and labelled partials.
  • Machine is a badge and a filter, never the session-list grouping. Grouping would destroy the attention ordering from c98755f. Staleness is the hard part: a status renders live only if the machine reporting it is live.
  • Federation requires ZeroID; refused under --local. Follows the Telegram precedent — refused once at construction, no if (localMode) in the verified path, per the invariant local-mode.md tells reviewers to enforce.
  • Attestation is an existing ZeroID extension point, not a new spec. ProofTypeTPM, AttestationLevel, and RequiredTrustLevel already exist. The tpm verifier is stub-only and ZEROID_ALLOW_UNSAFE_DEV_STUB defaults on, so this ships as an L0/L1/L2 ladder and stays honest about which rung it is on. WIMSE's attestation draft is TEE-only and does not cover this hardware; SPIRE node attestation is the closer model.

Prior art (scanned Sep 2026)

happy, Omnara, Vibe Kanban, Sculptor, Coder Agent Relay — all do remote control of one machine; none aggregates several. Omnara answers the same need with the opposite primitive (migrate the session to the cloud), and is documented as the rejected alternative with the reasons it loses here: work pinned to a machine by Apple signing, the corpus, or an unpushed branch.

Sequencing

Federation does not jump the conductor's critical path. P5 front doors (#279) stay first — a search of web/src/ and src/frontends/ finds no consumer of FleetSnapshot at all, so nothing draws the fleet board today.

Review asks

  1. The retrieval split (§7) — is leaving episodes on their machine right, or should the hub hold everything?
  2. The rejected alternative (§11) — does centralising execution actually lose for this fleet?
  3. Sequencing (§12) — P5 before F0 is the load-bearing claim; cheaper to argue now than after F0 lands.
  4. §8 stale reclaim is unsolved and marked blocking for F3, not for the read-only board.

Note: POSITIONING.md self-describes as "untracked, not published" but is in fact tracked. That header was already inaccurate; flagging rather than fixing it here.

🤖 Generated with Claude Code

saucam and others added 2 commits September 5, 2026 13:38
The conductor is designed for one daemon. FleetSessionView carries no
machine (fleet.ts:33), machine_map derives workspaces from the local
session list (fleet.ts:314), and sessions/dispatch_tasks/episodes have
no host column — so "show me every session across my machines" is not
merely absent, it is inexpressible. No open issue covers it.

Adds docs/federation-design.md proposing a machine axis plus a hub and
outbound-dialing satellites. Four pieces already exist and carry most of
the weight: resolveWorkdirAlias() (share/git-alias.ts) already computes
the cross-machine join key, path-rewrite re-anchors paths, dispatch_tasks
is already a distributed queue running in one process, and the
fleet.subscribe schema comments that widening `scope` past "tenant" was
anticipated.

Decisions worth flagging:

- Mirror session cards to the hub, leave episodes on their machine, and
  scatter-gather for deep recall with a deadline and labelled partials.
  The corpus stays where it was produced.
- Machine renders as a badge and filter, never as the session-list
  grouping — grouping would destroy the attention ordering from c98755f.
  Staleness is the hard part: a status renders live only if the machine
  reporting it is live.
- Federation requires ZeroID and is refused under --local, following the
  Telegram precedent rather than adding an if (localMode) branch to the
  verified path. Local-mode and POSITIONING updated to agree.
- Machine attestation is a ZeroID extension point that already exists
  (ProofTypeTPM, AttestationLevel, RequiredTrustLevel), not a new spec.
  The tpm verifier is stub-only today and the unsafe dev stub defaults
  on, so this ships as a trust ladder and stays honest about which rung
  it is on.

Prior art scanned Sep 2026: happy, Omnara, Vibe Kanban, Sculptor and
Coder Agent Relay all do remote control of one machine; none aggregates
several. Omnara's session migration is the opposite primitive and is
documented as the rejected alternative, with the reasons it loses here.

Design only — no code, and nothing draws the fleet board yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Converts the four open review asks into decisions, and fixes the one
that was a genuine design gap rather than a question.

§8 stale reclaim — resolved, not deferred. The hazard was that
claim_owner is a boot id, so #reclaimStale (dispatch.ts:407) treats any
foreign claim or expired lease as a crash to retry; federated, that
could re-run a half-finished ship elsewhere. Fixed by machine affinity:
only the owning machine may claim its task, so reclaim means "return to
queued for the same machine" and cross-machine re-execution is
impossible by construction — no liveness heuristic needed. On return a
satellite reconciles rather than re-runs (dispatch.ts already writes
worker_session_id at claim time and tolerates a surviving worker), and
shape decides the care taken: scout re-runs freely, ship blocks for
review when its worker is gone.

§8.2 records a bug federation would otherwise have introduced: lease
expiry currently means "the runner crashed", so a laptop closed
overnight would burn failureLimit on a task that never failed and
auto-block good work. Tasks now park while their machine is offline —
lease released, attempts untouched — so failureLimit keeps meaning
"this task keeps failing" rather than "this machine keeps sleeping".

The other three asks are locked with the reasoning that would let a
reviewer overturn them:

- Retrieval (§7) stays scatter-gather, with a named fallback instead of
  an open question: if F2 misses the 2s budget, degrade one machine at a
  time via per-machine `replicate: cards | full`, not by abandoning the
  architecture.
- The centralise-instead alternative (§11) is rejected on this fleet's
  specifics — Apple signing, the corpus, unpushed branches — and the doc
  says which of those changing would make it the better design.
- Sequencing (§12) keeps P5 first on a falsifiable claim: nothing
  consumes FleetSnapshot today, so F0/F1 would build an aggregation
  nothing can display.

Open questions 1-10 now all carry recommendations. Two remain genuinely
open because they need measurement rather than a call: the retrieval
fallback trigger, and the credential-lifetime number that §13.2 and
item 10 both hang on.

Also corrects the POSITIONING.md header, which claimed to be untracked
while being tracked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@saucam

saucam commented Sep 5, 2026

Copy link
Copy Markdown
Collaborator Author

Review asks resolved (ba87952)

All four are now decided in the doc rather than left as questions. One was a genuine design gap and is fixed; three are locked with the reasoning that would let a reviewer overturn them.

1. §8 stale reclaim — solved, not deferred

This was the real gap. claim_owner is a boot id, so #reclaimStale() (dispatch.ts:407) treats any foreign claim or expired lease as a crash to retry — federated, that could re-run a half-finished ship on another machine.

Fixed by machine affinity rather than a liveness heuristic. Only the owning machine may claim its task, so reclaim means "return to queued for the same machine" and cross-machine re-execution becomes impossible by construction. The hub never has to decide whether the laptop is dead or partitioned, because the answer doesn't change who may execute.

On return the satellite reconciles rather than re-runsdispatch.ts already writes worker_session_id at claim time and already tolerates a surviving worker across restart. shape then decides the care taken:

shape live worker on return no worker
scout adopt re-run (attempts++)
ship adopt block for review, never silently re-run

2. §8.2 — a bug this would otherwise have introduced

Worth flagging separately. Lease expiry currently means "the runner crashed", so attempts++ and failureLimit (default 2) auto-blocks after a couple of ticks. Federated, a laptop closed overnight would burn the budget on a task that never failed and auto-block good work with an error of "stale claim" describing nothing.

Tasks now park while their machine is offline — lease released, attempts untouched, shown as parked not failed — so failureLimit keeps meaning "this task keeps failing" rather than "this machine keeps sleeping".

3. Retrieval (§7) — locked, with a trip-wire

Stays scatter-gather. Instead of an open question it gets a named fallback: F2's exit criterion is the test, and if p95 misses the 2s budget the answer is per-machine replicate: cards | full — mirror the laptop that is usually asleep, leave the always-reachable Hetzner box in place. Degrades one machine at a time rather than collapsing the architecture.

4. Centralise-instead (§11) and sequencing (§12) — locked

A is locked on this fleet's specifics (Apple signing, the corpus, unpushed branches), and the doc names which of those changing would make B the better design — so it can be revisited rather than defended.

P5 stays first on a falsifiable claim: nothing consumes FleetSnapshot today. The one-line test for anyone wanting to reorder — if the board had a consumer, F0 would go first.

Open questions

All ten now carry recommendations; items 1, 2, 4, 6, 7, 8, 9, 10 are marked Decided with rationale, 3 and 5 resolved by other sections. Two stay genuinely open because they need measurement rather than a call: the retrieval fallback trigger, and the credential-lifetime number — item 10 makes it the single knob for both attestation freshness and the §13.2 outage window, so it is worth choosing deliberately.

Also

POSITIONING.md's header claimed "untracked, not published" while being tracked — corrected, and the stale date bumped.

🤖 Generated with Claude Code

Surveys the specs this design could follow, before writing code. The
most valuable finding is one to reject.

SPIFFE Federation is the wrong spec, despite the name. It exists so
SEPARATE trust domains — different orgs or clusters — can exchange trust
bundles through bundle endpoints, one relationship per domain. These
machines are one owner, one ZeroID tenant, one administrative boundary:
nodes in a single trust domain, not domains that federate. Adopting it
would model four of the owner's own computers as four organisations, with
N bundle endpoints and N trust relationships, to solve a mutual-distrust
problem that does not exist. The correct analogue is node attestation
within one trust domain — plain SPIRE, not SPIRE federation. This also
re-confirms open question 6 (machine is an attribute, not a tenancy key)
on a structural argument rather than a convenience one.

Hybrid Logical Clocks are adopted, and this is the one finding with an F0
consequence. §9 previously ordered cross-machine events by hub receipt
time, which diverges from causal order exactly during the batched,
reconnect-driven replication that makes ordering matter. HLC is constant
space (unlike vector clocks), preserves causality under skew, and stays
close to wall-clock so the bi-temporal "state last Tuesday" queries keep
working. The hlc column lands in F0 because a causal order cannot be
reconstructed after rows exist on several machines.

Also folded in:

- RFC 9334 (RATS) vocabulary; ZeroID's submit-verify-carry flow is
  already the Passport model. Its §10 epoch IDs answer open question 10
  better than the credential-renewal trigger alone did — epoch IDs are
  explicitly designed to work without trusted time, which is the thing
  §9 says we lack. The two compose: renewal is when, epoch ID is how.
- IEEE 802.1AR DevID for the L2 path, specifically LDevID (owner-
  provisioned) since consumer laptops rarely ship manufacturer IDevIDs.
- A2A v1.0 rejected for the hub link — it solves cross-org, cross-vendor
  delegation, and both ends here are codeoid sharing a typed protocol —
  but it is the right spec for the external surfaces already tracked in
  #61 and #251. Its Agent Card concept is worth borrowing as a
  per-machine capability blob, which routing needs anyway ("build the
  iOS target" must land on the Mac).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@saucam
saucam merged commit 1e68115 into main Sep 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants