Add resume/recover loop: never restart an issue from scratch - #36
Merged
Conversation
Turns fleet_dispatch from one-shot ("try an issue once, PR or bust") into a
loop that moves forward across attempts, per the design agreed with the user:
agent-driven, triggered by stalled dispatch outcomes, all three parts.
A. Durable attempt state. Every terminal outcome now persists enough to resume:
the branch is pushed even on failure/partial (not just success), and the
dispatch ledger entry carries outcome, attempt number, commit counts, pushed
flag, blocker ref, and the worker's final message. Outcomes are now the full
set: success / needs_review / no_changes / failed / blocked / needs_human.
B. Resume from the prior attempt. main() reads each issue's last attempt
(_last_attempt) and routes via a pure _dispatch_decision: fresh / resume /
skip. A resumed dispatch checks out the prior pushed branch (base
origin/<branch>) instead of main, and the prompt hands the worker the prior
outcome + final message and tells it to `git log/diff` what's there and
continue, not restart.
C. Blocker -> cross-repo issue. When an issue is blocked by a missing capability
in another repo, the worker files an issue there (gh issue create, now
allowlisted) and prints a `FLEET-DISPATCH-BLOCKED: <org>/<repo>#<n>` sentinel;
the tool records outcome=blocked and skips the issue until _issue_is_open
says the blocker closed. After MAX_ATTEMPTS (3) unresolved tries an issue is
marked needs_human and left alone.
Commit-count uses merge-base so it's robust to main advancing under a concurrent
dispatch. +13 tests (blocker parse, decision table, attempt lookup, resume
prompt, main routing). 47 tests, ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The live #17 run exposed this: both accounts hit their session limit ("resets 6pm"), so two attempts recorded 'failed' -- and under the plain MAX_ATTEMPTS cap a third transient hiccup would have dumped a perfectly-fine issue to needs_human. A rate limit is not the issue's fault. - New 'deferred' outcome for a transient/infrastructure failure (session/rate limit, overloaded, service unavailable, matched narrowly in the worker's final message). It's resumable but excluded from the attempt count, so a string of rate limits resumes indefinitely without ever escalating. - _last_attempt's cap count self-heals: it also ignores any 'failed' entry whose message reads transient -- including the two #17 failures already in the live ledger, recorded before this classification existed. - Resume-prompt wording fix (also from #17): only claim "you're on the branch it left behind" when a durable branch actually exists. A failed run that stalled before its first commit (a session limit) leaves nothing, so the prompt now says "starting from main" instead of promising an absent branch. _prompt_for takes has_branch, set from the real remote-branch check. 56 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The multi-account premise is silently void if two config dirs are logged into the same account -- they share one session window and quota, so "concurrency" buys nothing and one account drains twice as fast. This actually happened: ~/.claude-mythingslab and ~/.claude-lorenzoliuzzo were both authenticated as lorenzoliuzzo@outlook.com (same accountUuid), so every "account2" dispatch ran under lorenzoliuzzo and both hit its session limit -- invisible, because they're separate directories with separate tokens. _preflight_distinct_accounts reads each dir's accountUuid from .claude.json (oauthAccount, no token touched) and main() refuses to run -- dry-run or --execute -- if any two collide, or if an identity can't be read at all. Cheap, local, and it makes the footgun impossible to hit silently. 60 tests, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # fleet_dispatch.py # test_fleet_dispatch.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
These three commits were built in a prior session on this branch; branch already exists on origin, this just opens the PR that was never created for it.
Test plan
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com