Skip to content

fix(#565): captain re-adoption after cmux restart — role restoration + no-death-by-absence fail-safe - #568

Merged
tu11aa merged 3 commits into
developfrom
fix/565-liveness-readopt
Jul 11, 2026
Merged

fix(#565): captain re-adoption after cmux restart — role restoration + no-death-by-absence fail-safe#568
tu11aa merged 3 commits into
developfrom
fix/565-liveness-readopt

Conversation

@tu11aa

@tu11aa tu11aa commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Correction: this PR originally claimed the fix was a reconcileLiveness startedAt-ordering change. That theory was wrong (confirmed by the reporter against live daemon/cmux data) and does not explain the incident. It's kept as a defensive hardening only — see "Kept as hardening" below — but it is not what fixes #565.

The actual root cause and fix, verified against the live incident data pulled from the running daemon and the real ~/.cmuxterm/claude-hook-sessions.json:

cmux's own hook-sessions record for the still-alive bet2fun-app captain (pid alive 7h49m+) degraded to a bare launchCommand (["claude"], missing --append-system-prompt-file) after a crash/reattach. runLivenessTick's roleFromTemplate classification depends entirely on that flag, so the record read role:"unknown", was filtered out of the captain group, and the project fell out of seen on the very next tick — triggering markEnded()"stopped" forever, crews reaped, delivery permanently paused. The pid was alive the entire time; nothing ever re-checked that before inferring death.

Two changes, both in packages/core/src/daemon/delivery-loop.ts (runLivenessTick):

  1. Root cause fix — sessionId identity now outranks a degraded/unclassifiable launchCommand. A record whose sessionId matches an already-known captain for its project is treated as that captain regardless of what roleFromTemplate currently reads.
  2. Fail-safe / intentional contract change — a captain missing from the runtime snapshot is no longer marked ended on absence alone. markEnded now requires positive evidence: the tracked pid must be confirmed dead via isPidAlive. If the pid is still alive, or unknown/null, the entry is left alone (a log line is emitted instead). This also gates the Detect user-closed captain workspace → clean up (deregister relay, reap orphaned crews) + set explicit 'stopped' status #324 reaper, which fires off the same derived state — so live crews can no longer be reaped from an inferred signal.

This is a deliberate breaking change to the reap contract, not a side effect

Reaping is destructive — it terminalizes live, working crew tasks. The old contract ("absent from one runtime snapshot read → captain is stopped → reap its crews") is exactly what turned a cosmetic cmux store glitch into 3 cancelled crews that were actively committing code the morning of this incident. packages/cli/src/__tests__/squadrantd-daemon-direct.test.ts had a test that encoded the old contract directly (isPidAlive: () => true yet still asserting the crew got reaped on snapshot absence). Rather than loosen the fix to keep that test green, the test was updated to reflect the new, intentional contract:

  • The genuine-close case now simulates what actually happens on a real close — the snapshot goes empty and the pid dies — and still asserts the crew gets reaped. The real reap path is preserved and covered.
  • A new test reproduces the actual incident: captain absent from the snapshot but pid still alive → crew is not reaped, task stays "working".

Kept as hardening (not the fix)

packages/core/src/liveness.ts's reconcileLiveness: a live pid now outranks a startedAt-ordering comparison once prev is already dead, so a captain can't get stuck if reconcileLiveness ever sees a live reopen with an older startedAt for some other reason. Doesn't apply when prev is still alive (preserves the existing #527 stale-duplicate guard). Independently reasonable, doesn't fix #565 on its own.

Test plan

  • Added regression tests reproducing the exact incident: (a) a record with role:"unknown" but a known-captain sessionId is still treated as the captain, (b) a captain absent from the snapshot with a still-alive pid is NOT marked ended and NOT reaped (core-level + CLI integration-level), (c) same with an unknown/null pid
  • Updated the pre-existing tests that encoded the old (buggy) absence-implies-death contract — both in packages/core (unit) and packages/cli (integration) — to require genuine death instead, and added the inverse case alongside each
  • npx tsc -b on packages/shared/packages/core/packages/agents/packages/workspaces (leaf deps, needed to unblock local test resolution in this fresh worktree) + npx vitest run in packages/core (42 files/656 tests) and packages/cli (58/59 files, 479/480 tests — the one failing file is an unrelated pre-existing @squadrant/web unbuilt-dist issue in this worktree) — all green
  • npx tsc --noEmit clean in both packages/core and packages/cli

tu11aa added 2 commits July 11, 2026 22:15
reconcileLiveness's runtime/agent branch treated an incoming record
with an older startedAt than the stored one as stale and discarded it
outright — unless lastState was "end". That's correct when prev is
alive (guards against a stale duplicate session clobbering a live one,
#527), but wrong when prev is already stopped/gone: once a captain's
liveness entry is wrongly marked dead, no future runtime signal for
that same still-running session can ever win, because its startedAt
never changes and will keep losing the ">=" comparison forever. The
daemon then never re-adopts the captain, permanently pausing delivery.

Reproduced against the live incident data (packages/core/src/liveness.ts)
and added a regression test alongside a guard test for the #527 case.
…store role via sessionId

Root cause (corrected — the earlier reconcileLiveness startedAt theory in
this branch was wrong and does not explain the live incident): cmux's own
hook-sessions record for the live bet2fun-app captain degraded to a bare
launchCommand (`["claude"]`, no --append-system-prompt-file) after a crash/
reattach, so runLivenessTick's role-from-template classification read it as
role:"unknown". Filtered out of the captain group, the project fell out of
`seen`, and the very next tick markEnded() it — "stopped" forever, crews
reaped, delivery permanently paused, even though the pid was alive the whole
time (verified against the live cmux store data pulled from the running
daemon).

Two changes:

1. Root cause: sessionId identity now outranks a degraded/unclassifiable
   launchCommand. A record whose sessionId matches an already-known captain
   for its project is treated as that captain regardless of what
   roleFromTemplate currently reads.

2. Fail-safe (stands independently of #1): a captain missing from the
   runtime snapshot is no longer marked ended on absence alone. markEnded now
   requires positive evidence — the tracked pid confirmed dead via
   isPidAlive. If the pid is still alive, or unknown/null, the entry is left
   alone and a log line is emitted instead. This also gates the #324 reaper,
   which fires off the same derived state.

Updated the two existing tests that encoded the old (buggy) behavior
— "absent from snapshot marks ended/reaps even with isPidAlive() => true" —
to require genuine death, and added regression tests reproducing the exact
incident for both the role-restoration path and the fail-safe.
@tu11aa tu11aa changed the title fix(#565): reconcileLiveness re-adopts a live captain over a dead prev fix(#565): captain re-adoption after cmux restart — role restoration + no-death-by-absence fail-safe Jul 11, 2026
…act change

The daemon-direct reap test encoded the OLD contract: "captain absent from
the runtime snapshot" alone was sufficient to reap a project's orphaned
crews. That contract is what caused #565 — a single snapshot glitch (cmux
degrading a session's launchCommand) silently reaped 3 live crews that were
actively committing code. The previous commit changed the contract on
purpose: reaping now requires positive evidence the tracked pid is actually
dead, not just absent from one read.

- Updated the existing test's fixture to simulate a genuine close (the
  snapshot goes empty AND the tracked pid is confirmed dead) — this is what
  actually happens when a user closes a captain workspace, so the real reap
  path is preserved and still covered.
- Added the inverse case as its own test, reproducing the actual incident:
  captain absent from the snapshot but its pid still alive → the crew must
  NOT be reaped and its task stays "working".
@tu11aa
tu11aa merged commit 64c88f8 into develop Jul 11, 2026
1 check passed
@tu11aa
tu11aa deleted the fix/565-liveness-readopt branch July 11, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CRITICAL: once a captain workspace is reaped, the daemon never re-adopts it — delivery stays paused forever, with no reattach and no recovery path

1 participant