fix(performance): back off and bound heap-snapshot retries after a write failure - #734
Merged
Merged
Conversation
Refs #733 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
…ite failure On a failed writeHeapSnapshot the watchdog cleared its one-shot latch so the next sample could retry. Above 25% of the V8 limit the next sample is 2 s away, and the failure most likely at the trip line (ENOSPC on a disk full of debug artifacts, EIO) does not subside in 2 s, so the process repeated a multi-second synchronous snapshot write every 2 s — a freeze loop under exactly the condition this subsystem creates. A failed write now arms a ten-minute backoff; after three failures the watchdog gives up for the run and says so once. Success stays single-shot. Fixes #733 Refs #365, #364 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
Review follow-ups: a run that trips but can never write a snapshot left no durable trace because onHeapPressure only fired on success — it now fires on give-up with snapshotPath null and the last error, so the incident journal records the trip; each failed writeHeapSnapshot removes the truncated file it leaves behind so the backoff really frees the disk; the latch's states are documented and wall-clock backoff is deliberate; tests stub AGENT_CODE_HEAP_SUMMARY so no real child can spawn. Refs #733 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa
Owner
Author
|
Self-review pass (Fable adversarial review) — resolved in HEAD:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
src/main/performance/heapWatchdog.tswrites one synchronous heap snapshot when main'sused_heap_sizecrosses the trip line — a full GC plus graph serialisation, 3–15 s of hard freeze for the 190 MB dumps this machine produces, accepted as a one-shot forensic cost. On a write failure the catch block cleared the one-shot latch "so a later sample can retry once the immediate pressure subsides". Above 25% of the V8 limit the sampler runs every 2 s, and the failure most likely at that point (ENOSPCfrom a disk already holding 12 GB of debug artifacts,EIO) does not subside — so the process repeated a multi-second synchronouswriteHeapSnapshotevery 2 s until it died or the disk freed up. Details in #733.What changed
SNAPSHOT_RETRY_BACKOFF_MS(10 min); tripped samples inside the backoff are no-ops.MAX_SNAPSHOT_ATTEMPTS(3) failures the watchdog gives up for the run and logs that once, so the silence afterwards is explainable.__resetHeapWatchdogForTestsexposes a reset for the module-level state.Linked issues
Fixes #733. Refs #365, #364, #48, #103.
Verification
heapWatchdog.test.ts(node:v8,electron, paths and fs mocked; fake timers): one write on trip and none after; a failed write is not retried on the next 2 s samples but is after the backoff; three failures then no further writes for hours; a success after a failure firesonHeapPressureonce and stays single-shot; nothing below the trip line. 5 tests passed.npx tsc -p tsconfig.node.json --noEmit— clean.Plan:
docs/superpowers/plans/2026-09-03-heap-watchdog-retry-backoff.md.🤖 Generated with Claude Code
https://claude.ai/code/session_01Kk16MNVJtWAnCeGxGRuHqa