Skip to content

fix: stop a merge follow-up from blocking on the worktree that spawned it - #4227

Merged
atomantic merged 1 commit into
mainfrom
cos/task-mstolkdd/agent-8249579a
Aug 15, 2026
Merged

fix: stop a merge follow-up from blocking on the worktree that spawned it#4227
atomantic merged 1 commit into
mainfrom
cos/task-mstolkdd/agent-8249579a

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

A CoS merge follow-up (sys-rl-*) went straight to blocked with "Worktree creation failed — isolation was explicitly requested", orphaning the PR it existed to land. The task file showed the follow-up carrying useWorktree: true but no existingBranch, which made it look like a serialization problem. It wasn't — the server log has the real sequence:

18:00:12.783  Spawned merge follow-up sys-rl-… for PR #…
18:00:14.192  Worktree requested for sys-rl-… on existing branch cos/task-…/agent-…
18:00:15.558  fatal: 'cos/task-…/agent-…' is already used by worktree at '…/worktrees/agent-…'
18:00:16.297  Removed worktree for agent-…          ← 0.7s too late

cleanupAgentWorktree queued the follow-up before tearing down its own worktree. Git allows a branch in exactly one worktree, so the follow-up's git worktree add lost the race against a teardown that was already in flight. (The missing existingBranch was a consequence: updateTask strips the pointer on a terminal status.)

Fixes

1. Release the branch before queueing anything against it — on both producers in cleanupAgentWorktree: the PR path, and the if-missing net that hands a failed run's orphaned PR to the same follow-up machinery. The Copilot pre-request and the findPullRequestForBranch lookup still run first, since those need the checkout.

2. A branch-busy failure is now a timed pause, not a permanent block. The ordering fix closes the routine race, but a worktree removeWorktree refuses to delete (uncommitted changes) holds the branch until a human clears it. So agentWorkspacePrep classifies the failure via a new isBranchCheckedOutElsewhereError predicate and blocks with worktree-busy + a cooldownUntil — a new member of TIMED_COOLDOWN_BLOCKED_CATEGORIES, revived by the existing cooldown sweeper. Bounded at 5 waits (~10 min), after which it takes the ordinary worktree-failed block so the orphaned-PR card still reaches a human. The predicate is deliberately kept out of isGitLockError, whose retry budget is sized in milliseconds for bookkeeping locks.

3. A follow-up's own existingBranch survives a terminal status. updateTask stripped it as a spent resume pointer, but a follow-up sets that key as its configuration — it exists to land the PR on that branch. Re-running a blocked follow-up therefore cut a worktree fresh off main: the merge still targets the right PR by url, but any fix-and-push lands on a branch the PR never heard of. Now keyed on resumedFromAgentId, the marker the resume mechanism always writes alongside it — so the store stays task-kind-agnostic rather than naming one task type.

4. Two block reporters stay quiet on a self-reviving pause. The orphaned-PR notifier and the voice proactive trigger both dedupe once per PR/task, so announcing a cooldown would have swallowed the card for a real block the task lands on later.

Supporting cleanups: PAUSED_BLOCKED_CATEGORIES is now composed from TIMED_COOLDOWN_BLOCKED_CATEGORIES (the file's own anti-drift convention — every timed cooldown is a pause by construction), unblockExpiredOrphanCooldownsunblockExpiredCooldowns now that it sweeps the whole vocabulary, and worktreeBusyAttempts resets in reviveBlockedTask alongside the other spent budgets.

Test plan

  • server/services/worktreeManager.test.jsisBranchCheckedOutElsewhereError against both git wordings (is already used by worktree at / is already checked out at), and that it does not match the permanent '<path>' already exists / a branch named 'X' already exists failures or leak into isGitLockError.
  • server/services/cleanupAgentWorktree.test.jsremoveWorktree runs before addTask on both producers (asserted via mock.invocationCallOrder), and the Copilot pre-request still runs before the teardown that takes its checkout away.
  • server/services/agentWorkspacePrep.test.js — branch-busy pauses with worktree-busy + cooldownUntil and keeps existingBranch; counts up from a string-typed worktreeBusyAttempts; gives up at the cap with the git error in the reason; a permanent failure still hard-blocks.
  • server/services/cosTaskStore.test.js — a self-configured branch pointer survives a terminal block, one marked by resumedFromAgentId does not, worktree-busy keeps it, and reviveBlockedTask clears the attempt counter.
  • server/services/cosTaskGenerator.cooldown.test.js — the sweeper revives worktree-busy, preserving both the branch pointer and the attempt count.
  • server/services/orphanedPrNotifier.test.js — no card for a timed-cooldown block, and the check precedes the dedupe probe so the PR's one card isn't consumed.
  • Full server suite: 28 865 passing / 1381 files. (Three unrelated files intermittently hit the 10s test timeout on the loaded dev machine — a different set each run, all green in isolation.)

…d it

A PR's merge follow-up attaches its own worktree to the PR branch, and the
CoS evaluation tick preps that task a second or two after the cleanup queues
it — while that same cleanup is still tearing its own worktree down. `git
worktree add` then fails with "is already used by worktree at …", the task
is blocked as `worktree-failed`, and nothing is left to land the PR (the two
calls ran 0.7s apart in the reported incident).

Release the branch before anything is queued against it, on both producers in
`cleanupAgentWorktree` — the PR path and the `if-missing` net that hands a
failed run's orphaned PR to the same machinery.

That closes the routine race; a worktree `removeWorktree` refuses to delete
(uncommitted changes) can still hold the branch, so a branch-busy failure is
now a TIMED pause (`worktree-busy`) revived by the existing cooldown sweeper
rather than a permanent block, bounded at 5 waits before it gives up and takes
the ordinary `worktree-failed` block.

Two related fixes fall out:

- `updateTask` stripped `existingBranch` on any non-paused terminal status,
  including a follow-up's own copy — which is that task's configuration, not a
  resume pointer. Re-running a blocked follow-up therefore cut a worktree fresh
  off main, so a fix-and-push landed on a branch the PR never heard of. Keyed
  on `resumedFromAgentId` now, the marker the resume mechanism always writes
  with it.
- The orphaned-PR notifier and the voice proactive trigger stay quiet on a
  timed-cooldown block. Both dedupe once per PR/task, so announcing a pause
  that self-revives would have swallowed the card for a real block later.
@atomantic
atomantic merged commit 6aea73f into main Aug 15, 2026
7 checks passed
@atomantic
atomantic deleted the cos/task-mstolkdd/agent-8249579a branch August 15, 2026 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant