feat(run): durable halt-and-resume for waitable rate-limit windows — write resumeAt, release the lock, survive reboots

[admarble]

Motivation

#860 fixed the classification that kept --auto-wait (#804) inert, and recorded an explicit decision (docs/investigations/auto-wait-vs-halt-resume-860.md) to keep the in-process wait for now. That decision names two structural costs the in-process design cannot fix from inside:

• The per-issue lock is held for the entire wait (up to 5h). Nothing else can touch the issue; after a crash mid-wait, a stale lock whose PID is recycled reads as fresh (Nested phase claude that hangs after its turn is killed at ~106s, taking down the parent sequant run #856) and blocks the issue until manual locks clear or 6h age-based recovery.
• A reboot or power loss kills the run and buys nothing. The claude subprocess died at the failure, so the wait only ever purchases resumption, not continuity — after a reboot the phase restarts from scratch anyway, minus the run, plus a stale lock.

Halt-and-resume inverts both: exit cleanly at the window halt, record when to come back, and let re-entry (cheap since #837 skips completed issues) continue the run.

Proposed shape (from the #860 decision record)

1. resumeAt in issue state — written at the billing/window fail-fast (fix(run): -Q quality loop retries after an "Out of credits" exec instead of failing fast (billing cascades into "unparseable verdict") #799 path) when the payload carries a waitable window (isWaitableWindow, Five-hour window exhaustion is misclassified as terminal billing, so --auto-wait is inert on subscription plans #860). Machine-readable; sequant status shows "resumable at 07:00" instead of a bare failure.
2. Re-entry path — sequant run --resume (or equivalent) that no-ops until resumeAt, then re-runs the halted issues, leaning on fix(chain): --chain --ready-gate resume re-executes already-gated links (COMPLETED_STATUSES misses waiting_for_human_merge/blocked) #837's completed-issue skipping.
3. Lock hygiene — release the per-issue lock at halt; re-acquire on re-entry. Removes the Nested phase claude that hangs after its turn is killed at ~106s, taking down the parent sequant run #856-adjacent stale-lock exposure for waits entirely.
4. Re-entry bound — a counter mirroring AUTO_WAIT_MAX_WAITS so a window that never reopens cannot ping-pong a scheduler.
5. Scheduler recipe — ~20 documented lines of launchd/cron invoking the re-entry command; no daemon inside sequant.

Relationship to --auto-wait

Not a replacement initially. --auto-wait stays for the attended/overnight-laptop case (context preserved in memory, zero infrastructure). This issue serves the unattended case where reboot survival and lock release matter. If it proves strictly better in practice, deprecating the in-process wait is a later, separate decision.

Acceptance criteria

• AC-1: A run halting on a waitable window (per Five-hour window exhaustion is misclassified as terminal billing, so --auto-wait is inert on subscription plans #860 classification) with halt-and-resume enabled writes a machine-readable resumeAt to issue state and releases the per-issue lock before exiting.
• AC-2: The re-entry command is a no-op (clear message, exit 0) before resumeAt, and after resumeAt continues the halted issue without re-running issues already completed (fix(chain): --chain --ready-gate resume re-executes already-gated links (COMPLETED_STATUSES misses waiting_for_human_merge/blocked) #837 behavior).
• AC-3: Re-entries are bounded per issue; exceeding the bound halts with today's labeled terminal message.
• AC-4: A reboot between halt and re-entry loses nothing: state carries resumeAt, no lock is held, and re-entry proceeds normally.
• AC-5: Docs ship a copy-pasteable launchd/cron recipe for the re-entry command.

References

• Decision record: docs/investigations/auto-wait-vs-halt-resume-860.md
• Five-hour window exhaustion is misclassified as terminal billing, so --auto-wait is inert on subscription plans #860 (classification fix + AC-7 decision), feat(run): --auto-wait — continue a rate-limited run when the window resets instead of halting #804 (--auto-wait), fix(chain): --chain --ready-gate resume re-executes already-gated links (COMPLETED_STATUSES misses waiting_for_human_merge/blocked) #837 (resume skips completed issues), Nested phase claude that hangs after its turn is killed at ~106s, taking down the parent sequant run #856 (stale-lock/PID-recycling hazard), fix(run): -Q quality loop retries after an "Out of credits" exec instead of failing fast (billing cascades into "unparseable verdict") #799 (billing fail-fast)

[admarble]

Plan (spec)

Complexity: Standard (5 ACs, ~4 directories) · Scope: ⚠️ SCOPE_WARNING (Non-Goals section missing; all metrics green) → quality loop enabled.

AC Checklist

• AC-1: Waitable-window halt writes machine-readable resumeAt to issue state + releases the per-issue lock before exit — Unit + Integration
• AC-2: Re-entry command no-ops (clear message, exit 0) before resumeAt; after it, continues halted issues, skipping completed ones (fix(chain): --chain --ready-gate resume re-executes already-gated links (COMPLETED_STATUSES misses waiting_for_human_merge/blocked) #837) — Unit + Integration (injected clock)
• AC-3: Re-entries bounded per issue; exceeding the bound → today's labeled terminal message — Unit
• AC-4: Reboot between halt and re-entry loses nothing (state carries resumeAt, no lock held) — Integration
• AC-5: Docs ship a copy-pasteable launchd/cron recipe — Docs gate test (scoped + mutation-verified)

Implementation Plan

Phase A — halt side

1. src/lib/workflow/state-schema.ts: optional windowHalt: { resumeAt, phase, reentries } on IssueState (mirrors autoWait; counter co-located for atomic clearing).
2. src/lib/workflow/state-manager.ts: updateWindowHalt() / clearWindowHalt() — never-throw on untracked issues, like updateAutoWait.
3. src/lib/workflow/batch-executor.ts: at the halt sites, when isWindowHalt(result) (not billing — credits aren't waitable), write resumeAt = resetsAtToMs(resetsAt) + AUTO_WAIT_BUFFER_MS. Clear on fresh issue start / completion.
4. Lock release needs no new code — run-orchestrator already registers releaseAll() on shutdown + process.on("exit"); verified by an integration test asserting no lock file remains after a halt.

Phase B — re-entry
5. New sequant resume command: no-arg scan of state for windowHalt issues; pre-resumeAt → no-op exit 0; post → bound check (MAX_RESUME_REENTRIES = 2, mirroring AUTO_WAIT_MAX_WAITS), increment counter, delegate to the existing run entry with resume semantics (phase markers + #837 skipping). Locks re-acquired via the normal run path.
6. sequant status (+ MCP status): render ⏸ Halted — resumable at <time> (re-entries N/2).

Phase C — docs
7. Recipe in docs/reference/ (cron line + launchd plist invoking npx sequant resume), cross-linked from docs/investigations/auto-wait-vs-halt-resume-860.md.

Key decisions

• No new flag for the resumeAt write — it is pure bookkeeping on an already-halting path; the re-entry command is the opt-in. Rejected: overloading sequant run --resume (phase-marker semantics, requires issue numbers; cron wants a no-arg idempotent command).
• Counter persists in state, not the in-memory ledger — re-entries span processes by definition.

Open questions

1. Gate the write behind a flag? Recommendation: no (see above). Impact if wrong: one ignorable state field.
2. reentries reset point? Recommendation: cleared with windowHalt on fresh non-resume run or completion.

Recommended phases: spec → exec → qa (quality loop on; no-browser-test).

[admarble]

/qa Review — #892

AC Coverage — 5/5 MET

AC	Status	Evidence
AC-1 resumeAt written + lock released at halt	MET	recordWindowHaltState at both halt sites, gated on isWindowHalt (#860 classifier); billing/transient → null (unit-tested). Lock release via existing releaseAll() exit handlers (lock-manager.integration.test.ts).
AC-2 no-op before resumeAt; continues after, skipping completed	MET	Live: resume 101 → "⏸ not yet resumable…" exit 0; at-boundary counts as due; completed-status exclusion tested for all #837 vocabulary; delegates with resume: true.
AC-3 bounded re-entries + labeled terminal message	MET	MAX_RESUME_REENTRIES = 2 (mirrors AUTO_WAIT_MAX_WAITS), consumed before run, preserved across repeated halts; live: "Rate limited — re-entry bound reached (2/2)…" exit 1.
AC-4 reboot-safe	MET	Fresh-StateManager persistence test; live smokes from fresh processes; no lock survives exit.
AC-5 cron/launchd recipe	MET	docs/reference/halt-and-resume.md, gate test scoped to recipe markers, mutation-verified (payload deletion fails exactly that test).

Quality

• Type safety 0 issues · deleted tests 0 · security 0 critical · scope proportionate (15 files, +989/−2, ~44% tests)
• Build + lint clean; full suite 4913 passed (234 files)
• Execution evidence: help / dry-run / no-op / exhausted / empty-state all live-executed with captured exit codes
• CHANGELOG entry present; README + run-command.md + decision record cross-linked

Adversarial re-read (Gaps Found — recommendation-grade)

1. A re-entry is consumed even if the delegated run skips the issue because another session holds its lock (rare; costs 1/2 re-entries; recoverable via sequant run <N> --resume).
2. sequant status/MCP windowHalt display lines untested (mirrors autoWait precedent).
3. resumeCommand shell live-verified but not unit-tested (pure helpers are).

Verdict: AC_MET_BUT_NOT_A_PLUS

All ACs met; findings above are non-blocking improvements. Proceeding to PR.

[admarble]

/fullsolve Complete

Issue: #892
Status: AC_MET_BUT_NOT_A_PLUS (all ACs met; recommendation-grade notes only)
PR: #895

Summary

• AC: 5/5 met (AC-5 gate test mutation-verified)
• Tests: 4913 passed full suite + 28 new tests across state-manager / batch-executor / resume command
• Live execution evidence: all five sequant resume exit-code paths exercised against a synthetic halted-state file
• Quality checks: type safety 0, deleted tests 0, security 0 critical, scope proportionate

Iterations

• Exec: 1/3 · QA: 1/2 (no loop needed — verdict breaks to PR per policy)

Non-blocking follow-up notes (from QA)

1. Re-entry counter is consumed even if the delegated run skips a lock-held issue (rare, recoverable).
2. status/MCP windowHalt display lines untested (mirrors autoWait precedent).
3. resumeCommand shell live-verified but not unit-tested.

---

Ready for human review and merge.

[admarble]

QA follow-up gaps closed (commit 601c878 on PR #895)

All three notes from the QA review are now fixed:

1. Lock-skip no longer wastes a re-entry — sequant resume probes the per-issue lock (read-only LockManager.check) before consuming a re-entry; a lock-held due issue is skipped with a clear message and retried on the next scheduler tick. Doc updated in halt-and-resume.md.
2. Pause display lines tested — formatIssueState exported @internal; new status-format.test.ts pins the windowHalt line (phase, resumable-at time, sequant resume pointer, re-entry bound) plus the Five-hour window exhaustion is misclassified as terminal billing, so --auto-wait is inert on subscription plans #860 autoWait sibling and the no-record absence case. (MCP passthrough deliberately untested: it is a literal field copy of already-tested state, and testing it would require cwd manipulation in shared forked workers — reference_process_cwd_spy hazard.)
3. Command shell tested — 10 new tests drive resumeCommand via injected deps against a real StateManager on a temp state file: arg validation, every exit-code path, persisted re-entry consumption, dry-run non-consumption, lock-skip without consumption, subset runs, requested-issue filtering.

Build + lint clean; 25 tests in the two command test files, 157 in the targeted workflow suites; live CLI smoke re-verified.