feat: dispatch + routing — the conductor directs the fleet (P4) - #130
Merged
Conversation
Send-class fleet actions, safely. Four pillars: DURABLE WORK QUEUE (hermes Kanban). dispatch_tasks in the Store owns every dispatch's lifecycle — not the conductor's turn. Atomic single-statement claims (UPDATE..RETURNING), boot-id stale reclaim (a claim held by a dead boot = crash; attempts++ makes the reclaim counter double as the stuck-loop guard, auto-blocking at failure_limit), exponential retry backoff via a not_before gate, execute-at-most-once-per-tick invariant, lease renewal only while the worker is verifiably alive, per-tenant worker cap with anti-starvation kind filtering. Dispatcher loop mirrors the OAuth sweeper lifecycle; tick() is public so tests drive it without timers. R3 CONFIRM AS AN INVARIANT. fleet_send / fleet_spawn / fleet_interrupt are registered on codeoid_fleet but deliberately kept OUT of allowedTools (the SDK auto-allows listed tools and skips canUseTool entirely), so every dispatch rides the existing approvalId flow with the full input shown to the owner. On top, a HARD gate in #shouldAutoApprove/#peekAutoApprove: send-class fleet tools never auto-approve — not in autonomous mode, not under a budget. Both guards are mutation-tested. LEAF WORKERS. fleet_spawn creates disposable role:"worker" sessions with shape-capped identities: scout holds no tools:write, and NO worker ever holds session:* (it cannot see or direct the fleet). The token is a root grant sanctioned by the owner's spawn approval — deliberately NOT a conductor delegation, since ZeroID's scope intersection means the conductor's session-scoped chain can never carry tools:write (R1 working as intended); created_by records the conductor lineage. Workers run autonomous on a bounded tool budget; exhaustion wedges into waiting_approval, which the dispatcher surfaces to the conductor and resolves via lease expiry. DIGESTS, NEVER TRANSCRIPTS. Worker completion → bounded digest (final assistant message excerpt + episode summaries) → durable dispatch_events → ONE batched <fleet_events> injection into the conductor when it's idle (burst-collapse; events survive a crash between completion and delivery). Restart recovery: reclaimed spawn tasks continue their resumed worker session (re-arming the autonomous budget) or respawn fresh. Also fixes a pre-existing lost-approval race: the waiting_confirmation broadcast lands a beat before canUseTool registers its resolver, so a fast approve/deny in that window was silently dropped and the turn hung forever. Decisions arriving early are now buffered and consumed at registration. Tests: 45 new across dispatch-store (claim/reclaim/backoff semantics), dispatcher (lifecycle, crash recovery, cap, wedge, event batching), fleet send handlers, worker identity profiles, and the R3 gate (mutation- verified: removing either the hard gate or the once-per-tick guard fails the suite). 1110 unit tests + 6 live-ZeroID integration tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 40 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
📝 WalkthroughWalkthroughThis PR introduces a durable SQLite-backed dispatch queue (P4) enabling a conductor session to send prompts, spawn/continue worker sessions, and interrupt them via approval-gated fleet tools. It adds a Dispatcher engine with retry/backoff/failure-blocking, worker identity scoping, session role widening, hard-blocked auto-approval for send-class tools, and daemon lifecycle wiring, along with config, docs, and tests. ChangesFleet Dispatch Queue (P4)
Estimated code review effort: 5 (Critical) | ~100 minutes Sequence Diagram(s)sequenceDiagram
participant Conductor
participant FleetMCP as Fleet MCP Server
participant Dispatcher
participant Store
participant Worker as Worker Session
Conductor->>FleetMCP: fleet_spawn(prompt, workdir)
FleetMCP->>Dispatcher: enqueue(spawn task)
Dispatcher->>Store: dispatchEnqueue(task)
Dispatcher->>Store: dispatchClaimNext(bootId)
Dispatcher->>Worker: spawnWorker(prompt, shape)
Worker-->>Dispatcher: onSessionStatus(idle)
Dispatcher->>Worker: buildWorkerDigest()
Dispatcher->>Store: dispatchComplete(id, digest)
Dispatcher->>Store: dispatchEventAdd(task_done)
Dispatcher->>Conductor: deliverEvents(<fleet_events>)
sequenceDiagram
participant Owner
participant Session
participant Client
Session->>Session: shouldAutoApprove(fleet_send tool)
Session->>Client: waiting_confirmation tool call
Client->>Owner: display approval request
Owner->>Session: approve()/deny()
Session->>Session: waitForApproval resolves decision
Session->>Session: proceed or abort dispatch action
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #130 +/- ##
==========================================
+ Coverage 77.85% 79.49% +1.64%
==========================================
Files 89 90 +1
Lines 14395 15251 +856
==========================================
+ Hits 11207 12124 +917
+ Misses 3188 3127 -61
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
dispatcher.test.ts fakes the host; this adds the other half — the REAL SessionManager host with MockSessionProviders injected via a new test-only _testProviderFactory (mirrors SessionCreateOptions._testProvider): - spawn end-to-end: a real role:"worker" session with the autonomous budget, the sentinel-marked scout brief, a digest built from the worker's actual final message, teardown after completion, and ONE <fleet_events> injection into a real conductor session. - send end-to-end: conductor-attributed delivery with the owner-approved prefix; cross-tenant targets fail terminally (tenancy wall). - fleet dispatch deps exercised as real closures (enqueue lineage stamping, board mapping, workdir validation, cross-tenant interrupt refusal) via an extracted _fleetDispatchDeps — behavior unchanged. Test harness drains sessions before removing its temp dir so in-flight meta writes can't ENOENT into unrelated test files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/config.ts`:
- Around line 344-366: The dispatch configuration schema is missing
`retryBaseMs`, so `DispatchConfig.retryBaseMs` cannot be set through
`CodeoidConfig.dispatch` or overridden from env and remains stuck at the default
used by `Dispatcher`. Update `DispatchSchema` and the mirrored
`CodeoidConfig.dispatch` type in `src/config.ts` to include `retryBaseMs` with
the same default as `DEFAULT_DISPATCH_CONFIG.retryBaseMs`, and wire it into the
parsed `dispatch` block so `SessionManager` passes it through to `new
Dispatcher(...)`. Also add the corresponding `CODEOID_DISPATCH_*` environment
precedence handling for the new dispatch fields, matching the existing config
sections.
In `@src/daemon/dispatch.ts`:
- Around line 264-298: The spawn deferral logic in `#claimAndExecute` is too
broad: setting kindFilter to "send" after one tenant hits maxConcurrentWorkers
blocks all remaining spawn tasks across every tenant for the rest of the tick.
Update the scheduling in `#claimAndExecute` and, if needed, dispatchClaimNext to
scope deferrals to the specific offending (accountId, projectId) tenant instead
of the global kind, so other tenants’ spawn rows can still be claimed. Also
ensure dispatchRelease does not cause the same capped tenant’s deferred spawn to
monopolize the queue across ticks; use tenant-aware skipping or per-tenant
iteration to prevent starvation.
- Around line 300-353: The catch path in `#execute` leaves a worker session
orphaned when spawnWorker or continueWorker throws and dispatchFail returns
blocked or failed. Update the error handling in `#execute` to destroy the worker
via this.#host.destroyWorker for any task that already has task.workerSessionId
(especially after a failing continueWorker attempt), matching the cleanup
behavior used in `#finishWorkerTask`. Keep the existing dispatchFail and emitEvent
flow, but ensure the session is torn down before exiting the catch.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 337464c6-b675-4902-8b96-86b194bf8313
📒 Files selected for processing (16)
docs/conductor-design.mdpackages/protocol/src/types.tssrc/config.tssrc/daemon/agent-identity.tssrc/daemon/dispatch.tssrc/daemon/fleet.tssrc/daemon/server.tssrc/daemon/session-manager.tssrc/daemon/session.tssrc/daemon/store.tssrc/daemon/transcript.tssrc/tests/agent-identity-conductor.test.tssrc/tests/dispatch-store.test.tssrc/tests/dispatcher.test.tssrc/tests/fleet-approval-gate.test.tssrc/tests/fleet.test.ts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…own, retryBaseMs config - Replace the global kind='send' cap-deferral filter with per-task claim exclusion (json_each id list): a capped tenant defers only ITS task, so neither the sends behind it nor other tenants' spawns are starved. One mechanism now enforces both tick invariants (execute-at-most-once + no-head-of-line-blocking); test proves tenant B's spawn runs while tenant A sits at its cap. - Close ALL THREE worker-orphan paths on terminal spawn failure: the #execute catch (continueWorker throwing), the reclaim-to-blocked path (crash-looped to the limit with a resumed worker — found by the new test, beyond the reviewed site), and a partial spawn in the manager (session created but the brief send failed → destroy before rethrow). - Expose retryBaseMs through the config schema/type so the backoff is actually tunable, plus a CODEOID_DISPATCH_ENABLED env kill switch (other dispatch knobs stay file-config, matching the conductor block). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
P4 of the conductor (
docs/conductor-build-plan.md) — the phase that turns the read-only supervisor (P3, #124) into something that can safely direct existing sessions and spawn disposable workers. Four pillars:1. Durable work queue (hermes Kanban pattern)
dispatch_tasksin the Store owns every dispatch's lifecycle — not the conductor's turn — so tasks survive daemon restarts:UPDATE…RETURNINGwith a scalar subquery — two claimers can never take the same task.claim_owneris the daemon boot id; any claim held by another boot is a crashed run and gets reclaimed on the first tick. Every reclaim costs an attempt, so the reclaim counter doubles as the stuck-loop guard — a worker that keeps dying across restarts auto-blocks atfailure_limit.not_beforegate with exponential delay, plus an execute-at-most-once-per-tick invariant (without it, a failing task burned its whole failure budget inside one tick — caught by the tests).2. R3 confirm as an invariant, not a mode default
fleet_send/fleet_spawn/fleet_interruptare registered oncodeoid_fleetbut deliberately kept out ofallowedTools— the SDK auto-allows listed tools and skipscanUseToolentirely, so keeping them off the list is what makes every dispatch ride the existingapprovalIdflow with the full tool input shown to the owner. Zero new wire types.3. Leaf workers with shape-capped identities
fleet_spawncreates disposablerole:"worker"sessions. Identity profiles by shape: scout (investigate-and-report) holds notools:write; ship (deliver-a-change) does. No worker ever holdssession:*— it cannot see or direct the fleet (the leaf property).agent-identity.ts: the worker's token is a root grant sanctioned by the owner's spawn approval, deliberately not a conductor delegation — ZeroID's three-way scope intersection means the conductor'ssession:*-scoped chain can never carrytools:write, which is R1 working as intended.created_byrecords the conductor lineage for audit.waiting_approval, which the dispatcher surfaces to the conductor (the owner can attach and approve) and otherwise resolves via lease expiry.4. Digests, never transcripts
dispatch_events→ one batched<fleet_events>injection into the conductor when it's idle (burst-collapse: N completions = one wake). Events survive a crash between completion and delivery.Bonus fix: a pre-existing lost-approval race
The
waiting_confirmationbroadcast lands a beat beforecanUseToolregisters its resolver — an approve/deny arriving in that window was silently dropped and the turn hung forever. Found by the new gate test; fixed by buffering early decisions and consuming them at registration.Exit criteria (build plan P4)
fleet_find→fleet_sendbehind approval ✓Deferred (documented): per-workspace autonomy modes (v1 = always confirm, the safest default), heartbeat backoff supervision beyond the dispatcher tick,
.clean_shutdownmarker.Testing
45 new tests:
dispatch-store(claim/reclaim/backoff/tenancy semantics),dispatcher(send/spawn lifecycle, crash recovery, cap, wedge, event batching), fleet send handlers, worker identity profiles, and the R3 approval gate driven through a realSession+MockSessionProvider. Two mutation checks verified the load-bearing guards. 1110 unit tests, 6 live-ZeroID integration tests, typecheck + lint all green.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes
Tests