fix(runtime): bg-worker watchdog — revert process-gone runs' TODOs (#22 bg half) - #36
Merged
Conversation
…n reconcile (#22 bg half) The background half of #22: a bg worker that exits without a terminal record (host sleep, crash, pi restart mid-run) left its linked fleet TODO stuck `in_progress` forever. `reconcileRuns` already marked the run `aborted` in the durable RunLog + in-memory RunRegistry (v0.10.2, probe-driven, periodic via the widget's 60s timer + on session_start), but it never transitioned the TODO — the cross-session tracking entry stayed `in_progress` with no path to recovery. Fix: `reconcileRuns` now accepts an optional `todoSync: TodoSyncPort`. When it marks a run `aborted` (process-gone), if the run has a `todoId` (the RunLog meta carries it), it best-effort awaits `todoSync.markRunTodoReverted(todoId, undefined, "WORKER_EXITED_WITHOUT_RESULT: process gone (probe)")` — reverting a fleet-created TODO to `open` (retryable) + appending the diagnostic note. The link path only accepts open/in_progress TODOs, so reverting to `open` is the correct recovery for a linked one too (its prior was open/in_progress). The call is wrapped in try/catch (best-effort: the run is already marked aborted in the log + registry). `reconcileRuns` is now `async` (returns `Promise<string[]>` — it awaits the TODO transition). Callers fire-and-forget it so the session_start asyncRunner setup isn't blocked: - `index.ts` session_start: `void reconcileRuns(...).then(notify)`. - `fleet-widget.ts` 60s liveness timer: `void reconcileRuns(...).catch()`. - `FleetWidgetDeps` += `todoSync?`, wired in `index.ts` widget construction. Closes the bg-watchdog half of #22 (the foreground half was fixed by PR #30's EMPTY_RESULT guard). With both halves done, #22 can close. Tests: 4 new #22 tests in reconcile.test.mts — (1) process-gone run with a linked TODO → reverted to open + WORKER_EXITED note; (2) no todoId (track:false) → no todoSync call; (3) todoSync optional (back- compat); (4) failing todoSync (deleted TODO) is best-effort, run still aborted. Existing reconcile tests converted to async/await. 609/609 pass (was 605, +4); typecheck clean.
Address the PR #36 review minors: - Add EOF newlines to src/runtime/reconcile.ts + test/reconcile.test.mts (repo convention). - Notify wording: "TODOs reverted" → "linked TODOs reverted to open" so it doesn't over-promise for track:false runs (which have no todoId and thus no TODO to revert). The reconcile logic was always correct; only the notify text was imprecise. 609/609 pass; typecheck clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#22 — background worker-disappearance watchdog (the bg half)
Closes the bg half of #22. The foreground half (empty success from a
prompt()that resolves with no assistantmessage_end) was fixed by PR #30'sEMPTY_RESULTguard. With both halves done, #22 can close.Root cause (bg half)
A bg worker that exits without a terminal record (host sleep, crash, pi restart mid-run) left its linked fleet TODO stuck
in_progressforever.reconcileRunsalready marked the runabortedin the durable RunLog + in-memory RunRegistry (v0.10.2, probe-driven, periodic via the widget's 60s timer + onsession_start), but it never transitioned the TODO — the cross-session tracking entry stayedin_progresswith no path to recovery. (This was the exact repro in #22's comment: runfl-msb903yr, worker gone, worktree clean at base, fleet TODOtd-msb90400zllurjstuckin_progress.)Fix
reconcileRunsnow accepts an optionaltodoSync: TodoSyncPort. When it marks a runaborted(process-gone), if the run has atodoId(the RunLog meta carries it), it best-effort awaits:→ reverts a fleet-created TODO to
open(retryable) + appends the diagnostic note. The link path only accepts open/in_progress TODOs, so reverting toopenis the correct recovery for a linked one too (its prior was open/in_progress). Wrapped in try/catch — best-effort (the run is already marked aborted in the log + registry).reconcileRunsis nowasync(returnsPromise<string[]>). Callers fire-and-forget it so thesession_startasyncRunner setup isn't blocked:index.tssession_start:void reconcileRuns(...).then(notify).fleet-widget.ts60s liveness timer:void reconcileRuns(...).catch().FleetWidgetDeps+=todoSync?, wired inindex.tswidget construction.Tests
4 new #22 tests in
reconcile.test.mts:open+WORKER_EXITED_WITHOUT_RESULTnote.todoSynccall.todoSyncoptional (back-compat: callers not wiring it) — run still aborted.todoSync(deleted TODO) is best-effort — run still aborted, no throw.Existing reconcile tests converted to
async/await.Verification
pnpm typecheck— cleanpnpm test:run— 609/609 pass (was 605, +4)git diff --check— cleanCloses #22 (both halves now done).