Harden incident forensics: catch fast heap bursts, cap feed-debug, journal retention, detect prior-run OOMs

[Juliusolsson05]

Fixes #388. Related to #368, #370.

Why

The 2026-07-04 crash was a 15 s burst that took the main-process heap from ~40 MB to a V8 mark-compact abort at ~2.55 GB. Every preventive safety we had went silent:

• Heap watchdog trip at 3 GiB, sampled every 30 s → threshold never crossed, and even at 30 s cadence a sample straddling the burst was 50/50.
• feed-debug files reached 60–300 MB per session (verified from the crashed run's own disk state); the last 30 spans before death were dominated by ipc.handle.debug:append-feed-log at 55–867 ms each.
• debug-retention printed pruned 1 artifacts (30.9 MiB) reason=performance-append budget=13.8GiB only to console.warn; the always-on incident journal has no record.
• Prior-run classifier attributed the crash to force_quit_or_power_loss because it can't distinguish a V8 OOM from a hard poweroff.

What changed (6 files, ~370 lines)

src/main/performance/heapWatchdog.ts

• Trip threshold 3 GiB → 1.5 GiB (still capped by heapLimit × 0.70). Steady-state Agent Code main sits at 30–400 MB across 34-hour runs, so 1.5 GiB is ~3× the ceiling — quiet under normal use, catches both the 1.2 GB and 2.5 GB failure modes.
• Base sample interval 30 s → 5 s. Guarantees at least one sample during any ≥10 s burst.
• Adaptive fast-sample 0.5 → 0.25 of heap limit, so 2 s cadence engages before the new trip line instead of after it.
• Snapshot writes stay single-shot per run.

src/main/storage/debugRetention.ts

• New setDebugRetentionJournal(journal) sets a module-level sink.
• Successful prunes (removed > 0) and prune failures now journal?.record({ area: 'storage.retention', name: 'debug_retention.prune' | 'debug_retention.prune_failed' }), with reason, removed, bytesFreed, budgetBytes, scannedBytes, and ttlHours in data.
• No-op prunes stay silent — retention is called from many hot paths on a 5 min cooldown.

src/main/storage/feedDebugLog.ts

• New MAX_FEED_DEBUG_FILE_BYTES = 128 MiB. Per-session in-memory counter is primed once from stat() so it survives across restarts.
• When admitting a batch would cross the cap, we write ONE tombstone JSONL row ({sessionId, __feedDebugCapped: true, capBytes, fileBytesAtCap, droppedEntriesSoFar, ts}) and drop the batch instead of half-writing it. All further batches short-circuit and only advance the drop counter.
• Hard drop (not rotation): forensic readers see a clean end-of-file marker; the renderer's in-memory debug window is the source of truth anyway, so bounding the disk trail is safe.

src/main/incident/AppRunJournal.ts

• start() enables process.report.reportOnFatalError = true and sets process.report.directory to the run dir. V8 fatal aborts (OOM, ineffective mark-compact, FATAL ERROR) now emit a report.<ts>.<pid>.<seq>.json right next to events.jsonl with heap statistics, native/JS stacks, and env info.
• Defensive: throws around process.report access are caught so the flag can't gate journal start.

src/main/incident/previousRunClassifier.ts

• New classification main_oom_suspected.
• findNodeDiagnosticReport(priorRunDir) scans the prior run dir for report.*.json, reads a bounded 32 KB prefix, and extracts the top-level "trigger" field with a regex. OOMError and FatalError map to main_oom_suspected; this branch runs first, so a specific attribution wins over both JS-incident and minidump signals.
• Existing minidump-based fallback (fix(incident): correlate native crashes with Crashpad minidumps + adaptive heap sampling #364) unchanged for pre-reportOnFatalError runs.
• Evidence on the resulting app.prior_unclean_shutdown incident now includes nodeReportPath and nodeReportTrigger.

src/main/index.ts

• Wires the retention sink: setDebugRetentionJournal(appRunJournal) right after installProcessCrashHooks. The initial scheduleDebugStoragePrune('incident-run-start') inside AppRunJournal.start() kicks off async I/O; the .then handler that would journal a non-empty prune runs after setDebugRetentionJournal executes, so the wiring is race-free.
• Treats main_oom_suspected as crash-like severity (error) in the prior-unclean-shutdown incident routing.

Explicitly not in this PR

• IPC backpressure for debug:append-feed-log — the real root fix (bound main's in-flight-bytes counter and signal renderer to drop). Will land as a separate PR with a design note; that one has actual choices (drop vs. block, per-session vs. global).
• Retention bucket rebalancing — 22 % of ~13.8 GiB for feed-debug is generous but not obviously wrong. Now that retention actions land in the journal, real-world data can drive future adjustments.

Test plan

• npm run dev; verify ~/.config/agent-code/incidents/runs/<run>/ gets manifest.json, heartbeat, and events.jsonl as before, plus (on any prune) a debug_retention.prune event.
• Trigger a large prune (delete clean-shutdown marker on an old run and start Agent Code to force startup retention pass) and confirm the journal event appears in events.jsonl.
• Stream a synthetic ~150 MB of feed-debug entries into one session; confirm a tombstone line lands in feed-debug/<session>.jsonl and further appends stop.
• Force a V8 OOM (env: NODE_OPTIONS='--max-old-space-size=200' then run a heavy import) and confirm:
  • A report.*.json lands in the run dir with "trigger": "OOMError".
  • Next boot's incidents.jsonl records app.prior_unclean_shutdown with reason: "main_oom_suspected" and nodeReportPath in the context.
• Sanity-run one 20-agent orchestration session for 30 min to confirm the tighter watchdog + 5 s sampling doesn't fire under normal load.

🤖 Generated with Claude Code