Skip to content

Main process GC storm: 60Hz screen-snapshot pipeline allocates ~GB/s with concurrent live sessions #390

Description

@Juliusolsson05

Symptom

With ~10 live Claude sessions streaming, the main process pins ~80% CPU and the whole machine gets choppy (audio stutter, window-server jank). Main event-loop delay: mean ~25ms, p99 ~50ms. Downstream this causes repeated orchestration.prompt_delivery_failed — "did not confirm pasted prompt before submit (timeout)" incidents.

Diagnosis (2026-07-04, run 2026-07-04T07-58-52-287Z-main-71919-50e818)

This is NOT a retained-memory leak — it is allocation churn + a V8 major-GC storm.

Evidence:

  • A manual heap snapshot of the main process (forces full GC) contained only ~45MB reachable, while main.memory.rss metrics showed heapUsed oscillating 46MB → 1.2GB → 162MB → 1.1GB within seconds (see performance/runs/2026-07-04T07-58-52-main-71919/metrics.jsonl).
  • GC runs on background threads, so sample-ing the process shows the main thread idle in the run loop while the process burns 80% CPU.
  • Heap snapshot contains multiple ~1MB ANSI screen strings + xterm CircularList — the live serialization pipeline.

Root cause chain (per live Claude session, up to 60Hz)

  1. claude-code-headless src/terminal/HeadlessTerminal.ts scheduleFlush(): every PTY chunk arms a 16ms timer (snapshotIntervalMs ?? 16) that unconditionally builds four buffer serializations: plain (40 rows), markdown (40-row cell walk), recent (200 rows), recentMarkdown (200-row × 120-col cell-by-cell walk with row += chars string concatenation in terminalToMarkdown).
  2. src/ClaudeCodeHeadless.ts (~line 533): every snapshot then runs six parsers (trust/resume/permission/compaction/slash-picker/AskUserQuestion — two re-walk the xterm grid) and emits the snapshot three ways (screen, event w/ spread, screen-channel publish).
  3. agent-code src/main/sessions/forwarder.ts:34 ships all four strings over session:screen IPC every tick, regardless of pane visibility. The renderer has a no-op bail-out, but main pays the full allocation + serialization cost first.
  4. × ~10 concurrent sessions → hundreds of MB/s garbage → continuous major GCs → the CPU burn and event-loop jank.

Claude's TUI redraws ~60Hz whenever it is "working" (spinner), so this fires continuously even when nothing user-visible is changing.

Secondary findings (same snapshot, worth separate follow-ups)

  • 973 retained IncrementalSseParser + 977 TextDecoder instances (proxy adapters, claude-code-headless/src/proxy/) — lifecycle audit needed.
  • 858 live FSWatchers in main.
  • debug:append-feed-log consumed 58s of main-thread time over a 55-min run (known feed-debug firehose; also grew ~/.config/agent-code/feed-debug/ to 1.6GB).

Fix plan (branch in claude-code-headless submodule)

  1. Change-gate the expensive work: build cheap plain first; if unchanged vs the previous flush, skip markdown/recent serialization, parsers, and emission entirely.
  2. Slow the cadence: 16ms → ~100-250ms; this is a monitoring/parsing surface, not a display path.
  3. (Follow-up, agent-code side) visibility-aware session:screen forwarding so hidden panes don't get 4 strings per frame.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions