perf(worktree-activity): stream JSONL transcripts instead of readFile#87
Merged
Juliusolsson05 merged 2 commits intoMay 13, 2026
Merged
Conversation
The dominant source of transient memory spikes in the main process is
the `readFile(path, 'utf8') + split('\n') + map(JSON.parse)` pattern
used in several JSONL-reading sites. For a 50 MB transcript the
pattern transiently keeps a 50 MB Buffer, a 50 MB string, a 50 MB
array of substrings, AND the parsed JS objects all live at once.
This helper wraps the existing createReadStream + readline pattern
already used in ghostJournal.ts so callers can replace those whole-file
reads with a one-line-at-a-time async iterable. Peak transient memory
drops from O(file_size) to O(longest_line).
Used by the next commit in this branch. Future commits should be able
to reuse this in sessionIndex and historyLoader the same way.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
parseTranscriptForActivity used to do
`readFile(path) → text.split('\n') → map(JSON.parse)` per transcript.
For a 50 MB Claude transcript that transiently kept the 50 MB Buffer,
50 MB JS string, the split-array of substrings (~50 MB), and the
parsed JS objects all live at once. With WorktreeActivityIndex's
60 s background refresh calling this for each candidate sequentially,
the per-tick peak was ~100-200 MB — visible in the system-perf
popover as a recurring spike-and-drop pattern every minute,
suspected primary cause of recent main-process OOMs.
Switching to streamJsonl drops the transient peak to one line at a
time. Same output shape (IndexedTranscript), same malformed-line
behaviour (silently skip), same call site contract. Verified
test:worktree-activity passes unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 13, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 60 s background refresh in
WorktreeActivityIndexwas callingparseTranscriptForActivityper discovered transcript, and that function usedreadFile(path, 'utf-8').then(t => t.split('\n').map(JSON.parse)). For a 50 MB JSONL transcript that pattern transiently kept four representations live: the 50 MB Buffer, the 50 MB JS string, the 50 MB-ish array of substrings, and the parsed objects. Sequentially across all candidates per refresh, this matched the 100–200 MB spike-and-drop pattern visible in the system-perf popover every ~60 seconds and is the most likely primary cause of the recent main-process OOM crashes the user has been seeing during normal work.This PR:
streamJsonlhelper insrc/shared/runtime/that yields parsed JSONL objects one line at a time usingcreateReadStream+readline.createInterface(same patternghostJournal.tsalready uses).readFile + split + mapbody ofparseTranscriptForActivitywith afor awaitloop overstreamJsonl.Peak transient memory per parsed transcript drops from O(file_size) to O(longest_line) — typically tens of KB per line even for large tool_use entries.
Out of scope (separate PRs)
src/main/sessionIndex.ts:287(palette search) andsrc/main/sessions/historyLoader.ts:121(session resume + pagination). Both will reusestreamJsonlin follow-up PRs.worktree-activity-index.jsonfile inindexStore.tsis single-JSON, not JSONL — streaming it requires a format migration shim or a JSON-stream parser dependency. Worth doing later, deliberately deferred from this PR.Test plan
npm run test:worktree-activitypasses (unchanged from baseline)npm run test:review-fixespassesnpm run build:appsucceedsgit diff main | grep -ic cc-shellreturns 0AGENT_CODE_PERF=1 npm run dev, open badge popover, observelarge_object_spaceover ~5 minutes. The 60 s spike pattern that was hitting 100–200 MB should drop substantially (target: <30 MB per refresh).Stats
2 files changed (+78 / -11).
🤖 Generated with Claude Code