Skip to content

fix(performance): back off and bound heap-snapshot retries after a write failure - #734

Merged
Juliusolsson05 merged 3 commits into
mainfrom
fix/heap-watchdog-retry-backoff
Sep 3, 2026
Merged

fix(performance): back off and bound heap-snapshot retries after a write failure#734
Juliusolsson05 merged 3 commits into
mainfrom
fix/heap-watchdog-retry-backoff

Conversation

@Juliusolsson05

Copy link
Copy Markdown
Owner

Problem

src/main/performance/heapWatchdog.ts writes one synchronous heap snapshot when main's used_heap_size crosses the trip line — a full GC plus graph serialisation, 3–15 s of hard freeze for the 190 MB dumps this machine produces, accepted as a one-shot forensic cost. On a write failure the catch block cleared the one-shot latch "so a later sample can retry once the immediate pressure subsides". Above 25% of the V8 limit the sampler runs every 2 s, and the failure most likely at that point (ENOSPC from a disk already holding 12 GB of debug artifacts, EIO) does not subside — so the process repeated a multi-second synchronous writeHeapSnapshot every 2 s until it died or the disk freed up. Details in #733.

What changed

  • A failed write arms SNAPSHOT_RETRY_BACKOFF_MS (10 min); tripped samples inside the backoff are no-ops.
  • After MAX_SNAPSHOT_ATTEMPTS (3) failures the watchdog gives up for the run and logs that once, so the silence afterwards is explainable.
  • Success remains single-shot per run; trip line, adaptive sampling and the child-process auto-summary are unchanged.
  • __resetHeapWatchdogForTests exposes a reset for the module-level state.

Linked issues

Fixes #733. Refs #365, #364, #48, #103.

Verification

  • New heapWatchdog.test.ts (node:v8, electron, paths and fs mocked; fake timers): one write on trip and none after; a failed write is not retried on the next 2 s samples but is after the backoff; three failures then no further writes for hours; a success after a failure fires onHeapPressure once and stays single-shot; nothing below the trip line. 5 tests passed.
  • npx tsc -p tsconfig.node.json --noEmit — clean.

Plan: docs/superpowers/plans/2026-09-03-heap-watchdog-retry-backoff.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa

Juliusolsson05 and others added 3 commits September 3, 2026 12:58
Refs #733

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
…ite failure

On a failed writeHeapSnapshot the watchdog cleared its one-shot latch so
the next sample could retry. Above 25% of the V8 limit the next sample is
2 s away, and the failure most likely at the trip line (ENOSPC on a disk
full of debug artifacts, EIO) does not subside in 2 s, so the process
repeated a multi-second synchronous snapshot write every 2 s — a freeze
loop under exactly the condition this subsystem creates.

A failed write now arms a ten-minute backoff; after three failures the
watchdog gives up for the run and says so once. Success stays single-shot.

Fixes #733
Refs #365, #364

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
Review follow-ups: a run that trips but can never write a snapshot left no
durable trace because onHeapPressure only fired on success — it now fires
on give-up with snapshotPath null and the last error, so the incident
journal records the trip; each failed writeHeapSnapshot removes the
truncated file it leaves behind so the backoff really frees the disk; the
latch's states are documented and wall-clock backoff is deliberate; tests
stub AGENT_CODE_HEAP_SUMMARY so no real child can spawn.

Refs #733

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
@Juliusolsson05

Copy link
Copy Markdown
Owner Author

Self-review pass (Fable adversarial review) — resolved in HEAD:

  • Give-up not durably recordedonHeapPressure now fires on give-up with snapshotPath: null, snapshotError, snapshotAttempts; the journal records the trip. Test asserts one callback and exactly one "giving up" line.
  • Truncated snapshot survives a failed write → each failure rmSyncs the file (best-effort). Test asserts it.
  • Test hermeticityAGENT_CODE_HEAP_SUMMARY stubbed empty.
  • Latch states documented; wall-clock backoff stated as deliberate; plan refs aligned; test 4 annotated as a re-arm guard rather than a bug(performance): heap watchdog retries a synchronous snapshot write every 2 s after a write failure #733 regression test.

@Juliusolsson05
Juliusolsson05 merged commit 67dd2a6 into main Sep 3, 2026
2 checks passed
@Juliusolsson05
Juliusolsson05 deleted the fix/heap-watchdog-retry-backoff branch September 3, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(performance): heap watchdog retries a synchronous snapshot write every 2 s after a write failure

1 participant