Incident forensics hardening: catch fast heap bursts, cap feed-debug, journal retention, detect prior-run OOMs

[Juliusolsson05]

Context

On 2026-07-04 a 34h 15m Agent Code main process died from a V8 mark-compact abort at ~2.55 GB heap. Forensics from incidents/, performance/, and feed-debug/ reconstructed the timeline in ~5 minutes, but every preventive safety we had went silent.

What we saw

Fatal burst was 15 s, watchdog is 30 s:

07:58:03Z  heapUsed =    39 MB
07:58:08Z  heapUsed =   637 MB   (+600 MB in 5 s)
07:58:13Z  heapUsed =  1620 MB   (+1 GB  in 5 s)
07:58:18Z  heapUsed =  2174 MB   (last successful sample)
07:58:26Z  ABORT: Mark-Compact 2554 → 2554 MB, "last resort; GC in old space requested"

Watchdog threshold is min(3 GiB, heapLimit × 0.75) = 3 GiB on this box. Crash peaked at ~2.55 GB → never crossed threshold, so no heap.pressure incident and no heap snapshot were written.

The last 30 spans before death were dominated by ipc.handle.debug:append-feed-log (55–867 ms per call). Individual feed-debug JSONL files in the crash window ranged from 60 MB to 300 MB, with no per-file cap.

debug-retention printed pruned 1 artifacts (30.9 MiB) reason=performance-append budget=13.8GiB to console.warn but nothing landed in events.jsonl — the always-on journal has no trace of retention actions.

The next-boot classifier attributed the crash to the generic force_quit_or_power_loss because it can't distinguish V8 OOM from a hard poweroff.

What we're going to do (in this issue's PR)

1. Heap watchdog reconfig (heapWatchdog.ts)

   • Sample interval 30 s → 5 s so a 15 s burst is guaranteed to be seen.
   • Trip threshold 3 GiB → 1.5 GiB so we snapshot before we hit the wall (still single-shot per run; both the 2026-05-11 OOM at ~1.2 GB old-space and the 2026-07-04 burst at ~2.5 GB would now trip).
   • Related to and helps close Heap watchdog misses V8 old-space OOMs below global heap limit #368.

2. Journal debug-retention firings (debugRetention.ts)

   • Emit debug_retention.prune events into AppRunJournal on every non-zero prune so the always-on spine records the action, not just console.warn. Related to Persist crash-adjacent console breadcrumbs and structured OOM observability events #370.

3. Per-file cap for feed-debug (feedDebugLog.ts)

   • Hard cap of 128 MB per session's .jsonl. When exceeded, write a single tombstone line and drop further appends for that session. Retention keeps its bucket cap; this bounds a single pathological session.

4. Prior-run OOM detection (AppRunJournal, previousRunClassifier.ts)

   • Enable process.report.reportOnFatalError with directory pointing at the run dir so V8 fatal aborts write a diagnostic JSON alongside incidents.jsonl.
   • Classifier reads that report (plus the last heartbeat's heapUsed) so the next boot can file main_oom_suspected instead of force_quit_or_power_loss.

Explicitly out of scope for this PR

• IPC backpressure for debug:append-feed-log — this is the "root fix" and deserves its own PR with a small design note (drop vs. block, per-session vs. global counter). Filing separately.
• Retention bucket rebalancing — 22% of 13.8 GiB for feed-debug is generous but not obviously wrong. Revisit after we have journaled prune actions to base decisions on.

Success criteria

• A synthetic reproducer that streams ~200 MB of feed-debug JSONL in <15 s trips the watchdog and writes a .heapsnapshot.
• events.jsonl contains debug_retention.prune records after a prune runs.
• A killed-by-SIGABRT (or --abort-on-uncaught-exception simulated) run produces a Node.js diagnostic report in its run dir, and the next launch's classifier attributes it to main_oom_suspected.

Related

• Heap watchdog misses V8 old-space OOMs below global heap limit #368 heap watchdog threshold gap
• Persist crash-adjacent console breadcrumbs and structured OOM observability events #370 crash-adjacent breadcrumbs + OOM observability
• Heap: main-process OOM under heavy multi-agent orchestration — follow-up to #288 #327 main-process OOM under heavy orchestration
• Heap: evidence-driven second-pass optimization after incident collection #365 evidence-driven heap optimization