Web server OOMs within ~60s of any client connecting when a workspace contains a large session

[chstd]

Web server OOMs (V8 heap) within ~60 s of any client connecting when a workspace contains a large session

Environment

• @deepseek-ai/dsh 0.1.0-rc.7 via global npm install (also reproduced on 0.1.0-rc.6)
• Node.js v24.13.0, Windows 10 (x64), 32 GB RAM
• Start: dsh web (also with --trusted-host <non-loopback-authority>; the leak is identical with and without the flag)

Symptom

When any client connects — one real browser tab of the Web UI, or a single bare WebSocket to /api/events.mux — the server RSS grows ~150–170 MB/s until it hits the V8 heap cap and dies with FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory (exit code 134). Nothing is written to the server log before the crash; the client receives zero frames on events.mux.

Root cause (bisected)

The leak is triggered by a single large session (log ≈ 172k events, ≈ 3 MB zstd-compressed session.jsonl.zstd):

state	RSS after 30 s with 1 client
fresh server, no sessions	180 MB → 119 MB (stable)
all sessions except the large one (30+ sessions)	204 MB → 120 MB (stable)
with the large session present	183 MB → 3.9 GB → crash (7.6 GB @ 60 s)

Reproduces with real browser tabs and bare WebSockets, over localhost and behind a reverse proxy. The same server previously ran a full day with a connected tab while the session was smaller; the leak appeared once the session crossed a size threshold.

Reproducer

1. Start dsh web on a profile whose workspace contains one session with a large log (~170k+ events).

2. Wait for the UI to answer on the configured port.

3. Connect a single WebSocket:

   const ws = new WebSocket('ws://127.0.0.1:3080/api/events.mux');
   setTimeout(() => process.exit(0), 60000);

4. Watch the server RSS: grows ~150 MB/s until the OOM crash.

Impact

Any connected client kills the server within ~1 minute once a workspace contains a large session, making the Web UI unusable for long-lived sessions; two connected clients die faster.

Notes

• Session logs live under $DSH_HOME/sessions/... as zstd-compressed JSONL.
• Telemetry is disabled by default (DSH_TELEMETRY_MODE unset) — unrelated.
• Happy to provide logs/heap profiles on request.

[Homoe-hs]

dsh web OOM investigation

1. Scenario: Linux reproduction matching #3275

• Environment: Node v26.7.0, Linux x86_64, 27GB RAM, 32 cores, KDE Plasma (Wayland + X11)
• Version: @deepseek-ai/dsh / dsh-web-app 0.1.1-rc.1, launched via pnpx @deepseek-ai/dsh web
• Symptom: the server starts fine (prints dsh web: http://127.0.0.1:3080, attempts to open the browser), then dies ~38s later:

<--- Last few GCs --->
38026 ms: Scavenge (during sweeping) 4087.1 (4092.7) -> 4084.0 (4093.2) MB ... allocation failure
38695 ms: Incremental Mark-Compact (reduce) 4091.7 (4100.5) -> 4087.1 (4092.7) MB ... (average mu = 0.303, current mu = 0.274)
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Node's default --max-old-space-size resolves to 4GB on this machine (4092MB in the GC log), the process aborts with SIGABRT and pnpm reports Command was killed with SIGABRT.

2. Heap snapshot evidence (live instance; 63,846,145 nodes / 3161MB live heap)

Type	Count	Size
plain Object	42,308,783	2192 MB
boxed heap number	14,056,022	225 MB
strings	—	348 MB
(object elements) arrays	78,299	345 MB

• Steady state process.memoryUsage(): rss 3.80GB / heapUsed 3.18GB / heapTotal 3.22GB; heap_size_limit 4,395,630,592 (≈4.09GB); total_allocated_bytes 31.6GB — a 3.2GB live heap behind 31.6GB of cumulative allocation (large amounts of short-lived objects)
• By name: Object 42,308,783 / 2191.8MB; system / Map 28,256; system / Context 44,239; largest single nodes are two ~110–119MB element arrays (session/message index-like containers)
• Strings in the heap contain historical model output and reasoning text — confirming the web process retains a fully parsed session object tree, not a disk file cache

3. The oversized session profile (session dir = 149MB compressed; one 8-day session alone = 115MB)

Decompressed and stream-analyzed the largest single session (protocol-level data only, no content):

• 115MB compressed (zstd, 253,590 frames) → 392.5MB JSON text, ratio ~3.4x
• 374,092 lines (JSONL, one event per line); event seq range 0 → 13,720,107 (internal event counter far above persisted lines, possibly related to compaction pruning)
• Time span: 8.26 days (2026-08-14 → 2026-08-22)
• Event mix by bytes:

Event type	Count	Bytes	Avg
assistant/message (assembled)	9,529	146.8MB	15KB
reasoning-chunks (delta)	168,936	100.2MB	0.6KB
assistant/chunk (delta)	85,886	49.8MB	0.6KB
tool-call-chunks (delta)	37,577	31.5MB	0.8KB
tool/code-dispatch + start	23,435	28.7MB	—
tool/result + tool/call	18,935	23.4MB	—
text-chunks	9,316	6.4MB	0.7KB
user/message	323	1.9MB	5.8KB
step/start + step/end	19,088	1.6MB	—

• Text volumes: reasoning chunks total 18.7M chars; assembled reasoning blocks 5,473 / 17.66M chars; tool-call argument blocks 9,432 / 8.39M chars; text blocks 4,843 / 1.05M chars; 323 user messages ≈ 1.09M chars
• Ratios: user : assistant messages ≈ 1 : 29 (9,529 assistant messages / 9,544 steps / 9,432 tool calls / 11,716 code dispatches)
• The session underwent 27 compactions (23 summaries / 71 prunes) — compaction did not stop history from being fully retained

4. Memory expansion mapping

• Session dir total: 149MB compressed → ~500MB decompressed JSON
• The 3.18GB snapshot = the parsed object tree of all sessions: ~5–6x text→object inflation (392MB text ≈ 2.2GB objects + 0.35GB strings + 0.22GB boxed numbers; remainder is the app itself)
• Conclusion: the web server decompresses, parses and retains every session at startup — heap usage grows linearly with the session directory, so any fixed cap will eventually be hit

5. Workaround verified & suggested next steps

Workaround verified: NODE_OPTIONS=--max-old-space-size=8192 runs fine (peak RSS ≈ 4.7GB / VmPeak ≈ 6.3GB), but only postpones the crash — usage still grows with session size (see #3560, which reports the same error text still ballooning to 6.1GB under a 6.3GB cap, though via a different root cause).

Suggestions:

1. Load sessions lazily/streaming in the web server (decompress one file at a time, retain only stats projections, release after use), or cold-store segments of very long sessions — rather than parsing all history into millions of retained objects.
2. At minimum: raise the heap cap from within dsh itself, or document the heap requirement (the 4GB default is guaranteed death for users with large session histories).
3. ~/.dsh/sessions has no usage hint / one-click archive UX — 149MB of session data is normal growth for a local tool, but there is zero feedback on the memory side.

Happy to share the full heap-snapshot aggregation (by type/constructor, 63.8M nodes) or the event-level stats script for validating any proposed fix.

[bcahlit]

Adding measurements from a Windows machine that reproduce this without any client, plus the retention point.

Environment: Windows 10 Pro 19045, Node v24.15.0 (default heap ~4.29 GB), 64 GB RAM.
@deepseek-ai/dsh@0.1.5-rc.1 CLI with 0.1.5-rc.2 subpackages (npm view @deepseek-ai/dsh version
returns 0.1.5-rc.1, so there is no newer release to test). Profile: web (dsh-base + dsh-web-app + a
community bundle).

1. Not client-triggered. dsh web --no-open — no browser, no WebSocket, no /api/events.mux —
reproduces identically: RSS climbs ~45–100 MB/s and dies in 60–80 s with the same
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory.
So a client connection is not required; boot-time session loading alone is enough.

2. The retention point. With --inspect attached:

• HeapProfiler.startSampling: >90% of live allocations are in consumeEventLine
  (packages/session/session-persistence-jsonl/lib/index.js:1079), the per-record JSON.parse.
• CPU profile stack: consumeEventLine <- SessionLogScanner.write:1030 <- readZstdPrefix:2792 <- decodeStoredLog:2650 <- requireStoredLog:2483.
• Debugger.setBreakpointByUrl on requireStoredLog fires 69 times in 8 s across 11 distinct
  sessions (one of them 17x), always via the persistence API open(id, "read").
• open(id, "read") returns a JsonlSessionHandle whose state carries primed: <fully decoded log>,
  so each open pins one whole decoded session in the handle tracker. decodeStoredLog also
  deep-freezes every record (freezeStoredEvents), and COLD_LOG_MEMO_MAX_ENTRIES = 2 means the
  decoded-log memo stops helping as soon as a couple of sessions are in play.
• Also hot: existsSync 11.5%, writeSync 5.0%, freezeStoredEvent 7.5%, GC 5.1%.

3. It is aggregate retention, not one bad or one large session. Keeping only part of the store:

session data present	RSS behaviour
none	flat ~400 MB
1 small session	flat 270 MB
the single largest session (5.0 MB)	flat ~278 MB
12 files / 3.9 MB	flat ~280 MB for the full minute
23 files / 12 MB	OOM
190 files / 159 MB	OOM in 60–80 s
A single session never triggers it, however large; it takes many. That fits "one retained decoded log
per session".

4. Ruled out (so nobody chases these):

• legacy v2 format — hiding all 147 session.jsonl.zstd and keeping only 23 v3 (12 MB) still OOMs;
• projection cache — moving storages/session_projcache aside makes it OOM faster (every session
  becomes a cold refold);
• third-party plugins — disabling all 21 rows of the community bundle via --patch cuts growth ~4x
  (75 -> 17 MB/s) but does not stop it;
• heap cap — --max-old-space-size=16384 still climbs past 4 GB with no plateau (matches the
  "only postpones the crash" conclusion above).

5. No corruption signal. dsh's corruption handling leaves artifacts (the projection cache moves
schema-invalid records aside as <id>.json.bak.<stamp>; persistence documents backup-and-skip).
There are none on this machine, all 141 projection-cache records parse, and across ~10 boots I never
saw a SessionPersistenceCorruptionError / SessionFormatUnsupportedError — only heap exhaustion.
(Caveat: I could not exhaustively parse the event frames — the file is a header frame plus further
zstd frames, and Node's public zstd decoder only yields the first — so this is behavioural evidence,
not a full file audit.)

6. Relationship to #5995. #5995 attributes unbounded growth to
packages/api/session-controller/src/client/sessions/manager.ts get() never evicting. My trace
instead pins the read side (open(read) -> primed). Both can hold simultaneously, and a plausible
single fix is for the session-controller to drop() sessions it no longer serves so the primed
logs are released. Worth a maintainer's look.

Candidate fixes on the read side: don't retain the decoded log in the handle state (primed);
bound/evict the per-session memo; or stream the fold instead of materialising every event.