Skip to content

fix(runtime): bg-worker watchdog — revert process-gone runs' TODOs (#22 bg half) - #36

Merged
rz1989s merged 2 commits into
mainfrom
fix/bg-watchdog-22
Aug 4, 2026
Merged

fix(runtime): bg-worker watchdog — revert process-gone runs' TODOs (#22 bg half)#36
rz1989s merged 2 commits into
mainfrom
fix/bg-watchdog-22

Conversation

@rz1989s

@rz1989s rz1989s commented Aug 4, 2026

Copy link
Copy Markdown
Member

#22 — background worker-disappearance watchdog (the bg half)

Closes the bg half of #22. The foreground half (empty success from a prompt() that resolves with no assistant message_end) was fixed by PR #30's EMPTY_RESULT guard. With both halves done, #22 can close.

Root cause (bg half)

A bg worker that exits without a terminal record (host sleep, crash, pi restart mid-run) left its linked fleet TODO stuck in_progress forever. reconcileRuns already marked the run aborted in the durable RunLog + in-memory RunRegistry (v0.10.2, probe-driven, periodic via the widget's 60s timer + on session_start), but it never transitioned the TODO — the cross-session tracking entry stayed in_progress with no path to recovery. (This was the exact repro in #22's comment: run fl-msb903yr, worker gone, worktree clean at base, fleet TODO td-msb90400zllurj stuck in_progress.)

Fix

reconcileRuns now accepts an optional todoSync: TodoSyncPort. When it marks a run aborted (process-gone), if the run has a todoId (the RunLog meta carries it), it best-effort awaits:

todoSync.markRunTodoReverted(todoId, undefined, "WORKER_EXITED_WITHOUT_RESULT: process gone (probe)")

→ reverts a fleet-created TODO to open (retryable) + appends the diagnostic note. The link path only accepts open/in_progress TODOs, so reverting to open is the correct recovery for a linked one too (its prior was open/in_progress). Wrapped in try/catch — best-effort (the run is already marked aborted in the log + registry).

reconcileRuns is now async (returns Promise<string[]>). Callers fire-and-forget it so the session_start asyncRunner setup isn't blocked:

  • index.ts session_start: void reconcileRuns(...).then(notify).
  • fleet-widget.ts 60s liveness timer: void reconcileRuns(...).catch().
  • FleetWidgetDeps += todoSync?, wired in index.ts widget construction.

Tests

4 new #22 tests in reconcile.test.mts:

  1. process-gone run with a linked TODO → reverted to open + WORKER_EXITED_WITHOUT_RESULT note.
  2. no todoId (track:false) → no todoSync call.
  3. todoSync optional (back-compat: callers not wiring it) — run still aborted.
  4. failing todoSync (deleted TODO) is best-effort — run still aborted, no throw.

Existing reconcile tests converted to async/await.

Verification

  • pnpm typecheck — clean
  • pnpm test:run609/609 pass (was 605, +4)
  • git diff --check — clean

Closes #22 (both halves now done).

rz1989s added 2 commits August 5, 2026 01:27
…n reconcile (#22 bg half)

The background half of #22: a bg worker that exits without a terminal
record (host sleep, crash, pi restart mid-run) left its linked fleet
TODO stuck `in_progress` forever. `reconcileRuns` already marked the
run `aborted` in the durable RunLog + in-memory RunRegistry (v0.10.2,
probe-driven, periodic via the widget's 60s timer + on session_start),
but it never transitioned the TODO — the cross-session tracking entry
stayed `in_progress` with no path to recovery.

Fix: `reconcileRuns` now accepts an optional `todoSync: TodoSyncPort`.
When it marks a run `aborted` (process-gone), if the run has a `todoId`
(the RunLog meta carries it), it best-effort awaits
`todoSync.markRunTodoReverted(todoId, undefined,
"WORKER_EXITED_WITHOUT_RESULT: process gone (probe)")` — reverting a
fleet-created TODO to `open` (retryable) + appending the diagnostic
note. The link path only accepts open/in_progress TODOs, so reverting
to `open` is the correct recovery for a linked one too (its prior was
open/in_progress). The call is wrapped in try/catch (best-effort: the
run is already marked aborted in the log + registry).

`reconcileRuns` is now `async` (returns `Promise<string[]>` — it awaits
the TODO transition). Callers fire-and-forget it so the session_start
asyncRunner setup isn't blocked:
- `index.ts` session_start: `void reconcileRuns(...).then(notify)`.
- `fleet-widget.ts` 60s liveness timer: `void reconcileRuns(...).catch()`.
- `FleetWidgetDeps` += `todoSync?`, wired in `index.ts` widget construction.

Closes the bg-watchdog half of #22 (the foreground half was fixed by
PR #30's EMPTY_RESULT guard). With both halves done, #22 can close.

Tests: 4 new #22 tests in reconcile.test.mts — (1) process-gone run
with a linked TODO → reverted to open + WORKER_EXITED note; (2) no
todoId (track:false) → no todoSync call; (3) todoSync optional (back-
compat); (4) failing todoSync (deleted TODO) is best-effort, run still
aborted. Existing reconcile tests converted to async/await.

609/609 pass (was 605, +4); typecheck clean.
Address the PR #36 review minors:
- Add EOF newlines to src/runtime/reconcile.ts + test/reconcile.test.mts
  (repo convention).
- Notify wording: "TODOs reverted" → "linked TODOs reverted to open" so
  it doesn't over-promise for track:false runs (which have no todoId and
  thus no TODO to revert). The reconcile logic was always correct; only
  the notify text was imprecise.

609/609 pass; typecheck clean.
@rz1989s
rz1989s merged commit 67fe35f into main Aug 4, 2026
1 check passed
@rz1989s
rz1989s deleted the fix/bg-watchdog-22 branch August 4, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

subagent: foreground runs can return empty success with no status, error, or run id

1 participant