Bug: dsh web V8 heap OOM after long uptime — live sessions are never unloaded

[steven9046]

dsh web: V8 heap OOM after long uptime — live sessions are never unloaded

Suggested labels: kind/bug, area/session, area/web.

TL;DR

Every session opened or resumed in the dsh web GUI becomes a permanently live in-memory Agent +
full event log; the AgentHandle.dispose() capability is dropped at open time, client disconnect
does not unload anything, and no idle/refcount eviction exists. Over weeks of GUI use the heap
crosses Node's 4 GB default cap and V8 aborts.

Environment

• dsh 0.1.0-rc.5, dev source launch: node --import tsx/esm apps/cli/src/bin.ts web (via pnpm dsh web)
• Node v22.23.2, Linux x64, machine uptime 31 days
• System RAM 125 GB with ~82 GB available at crash time — not a system-wide OOM

Crash

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Aborted (core dumped)
[ELIFECYCLE] Command failed with exit code 134

• Exit code 134 = SIGABRT from V8 after failing full GCs near the heap cap.
• Node's default heap_size_limit on this host (measured): 4144 MB.

Usage profile at crash time

• 11 workspace session stores under ~/.dsh/sessions/, 105 MB on disk (zstd JSONL); largest single
  session: 23,723 events / 21.8 MB decompressed JSON (robot_client workspace, still active).
• No single huge event: max event line 242 KB, only 8 events >100 KB — the heap was consumed by
  accumulation, not one allocation.
• The crashed instance had been running for a long time (machine up 31 days; exact process age not
  recoverable — no core was captured by apport for the V8 abort).

Observed RSS after restart (same workload resumed)

time (local)	elapsed	RSS
18:57	~1m	256 MB
19:00	~5m	488 MB

(Browser reconnects, restores session list, resumes the 23.7k-event session.)

Root cause (confirmed by code audit)

Live sessions are never unloaded in dsh web mode — every session ever opened or resumed in the
GUI stays fully in memory until process exit.

Evidence chain:

1. A browser RPC reaches ensureSession() (packages/host/apiproxy/src/api-proxy.ts:1618-1707),
   which resumes/creates the agent via ctx.agents.resume(...) / ctx.agents.create(...) and
   returns only .agent — the AgentHandle.dispose() capability is dropped
   (api-proxy.ts:1657-1661, 1670-1678).
2. AgentRegistry keeps one entry per session in a plain Map (packages/core/agent/src/index.ts:257);
   SessionStore likewise (packages/core/session/src/index.ts:792-793, plus the module-level
   attachments Map at :932). Removal happens only via detach()/dispose() or owner-fiber
   teardown — none of which web sessions ever get.
3. Each live Session holds its full event log in memory: private log: SessionEvent[] = []
   (packages/core/session/src/index.ts:425-428); "a resumed session's constructor seed is its full
   stored log" (index.ts:456-458).
4. WebSocket disconnect only aborts streams (packages/client/connection/src/websocket-downlink.ts:107);
   the persistence coordinator's retire() path is gated on session/disposed
   (packages/session/session-persistence/src/coordinator.ts:1132, 1140-1161), which never fires for
   web sessions. The coordinator itself documents the posture: "State created through the public
   create()/load() API has no owner" (coordinator.ts:230-234).
5. No idle-TTL / refcount / eviction mechanism exists anywhere in the web path (greps over apiproxy,
   core/agent, core/session, session-persistence). The only bounded structure is the prepared-session
   LRU (DEFAULT_PREPARED_SESSION_CACHE_SIZE = 5, coordinator.ts:27, 609-623), which bounds
   unpublished preparations, not live sessions.

Measured cost: parsing the 23.7k-event session log in isolation costs ~49 MB heap in plain
JSON.parse — before the agent loop, projections, surface manager, and per-client fan-out multiply it.

Observed RSS after restart (same workload resumed) — update

time (local)	instance	elapsed	RSS
18:57	2nd	~1m	256 MB
19:00	2nd	~5m	488 MB
19:02	—	—	2nd instance replaced after ~7 min (no core captured)
19:55	3rd	~53m	813 MB
10:05 (+1d)	3rd	15h	974 MB

Disk growth of the active session over the same spans: 7.49 → 7.96 MB (first hour), 8.96 MB by next
morning — server RSS grows far faster than session data itself. Growth decelerated overnight while no
new sessions were opened (813 → 974 MB over 14 idle hours), consistent with the per-opened-session
accumulation model above rather than a time-based leak: the crashed instance had weeks of GUI
browsing across 11 workspaces to reach the 4 GB cap.

Contributing factors (full audit, ranked)

1. Cold-session resume pins everything forever (primary, matches the multi-GB growth curve).
   packages/api/remotes/src/agent-lookup.ts:142-186 resumes every cold session the GUI touches
   into a permanent AgentRegistry entry; each entry = full in-memory event array + one Cordis
   agent fiber with the preset's whole plugin set + projection cells (WeakMap<Session, UnitCell>,
   packages/session/session-projection/src/index.ts:150 — GC-eligible only when the Session is
   unreachable, which never happens) + title work + telemetry adopted Set
   (packages/session/session-telemetry/src/coordinator.ts:66, deleted only on session/disposed).
2. Preparations LRU is count-bounded but size-unbounded — preparations.ts:33, 285-298 with
   coordinator.ts:244-252: up to 5 ready entries, each holding a full parsed Session and a
   frozen inspection referencing the same events. Browsing the 22 MB session leaves its parsed log
   pinned in the LRU after the request returns; eviction is correct but the working-set floor is
   large and GC pressure matches the "Ineffective mark-compacts" signature.
3. Unbounded per-client FrameQueue on the all-sessions firehose — api-proxy.ts:413-437
   (private buffer: F[] = [], push with no high-water mark); the per-connection session/event
   listener (api-proxy.ts:3475-3489) pushes every committed event of every session to every
   connected client — no per-session filter, no backpressure. Clean aborts are handled
   (:3530-3533), but a slow/stalled client (half-open TCP, paused reader) accumulates without
   bound while its stream stalls. Grows with (events × clients).

Audited and cleared: subagent children (disposed after run,
packages/subagent/subagent-in-process-driver/src/index.ts:195-199), workflow workers
(worker.terminate(), packages/workflow/workflow-worker-thread/src/host.ts:200,244), todo/plan/
compaction state (WeakMap on Session lifetime), projection-cache (disk-backed), history replay
(re-reads per request, no cache), apiproxy dedupe maps (deleted after settle).

Ask

1. Wire an unload path for web-opened live sessions: idle-TTL or client-refcount eviction that calls
   the currently dropped AgentHandle.dispose() (packages/core/agent/src/index.ts:159-174) so the
   session/disposed → retire() chain (coordinator.ts:1132-1161) actually runs.
2. Cap the mux FrameQueue with a high-water mark + drop/backpressure policy
   (api-proxy.ts:413-437), and consider per-session subscription filters.
3. Bound the preparations LRU by bytes as well as count, or drop the frozen inspection reference.
4. Make the eviction budgets Config fields (cordis.yml) and document the memory model for
   long-running dsh web deployments.
5. Short-term stopgap in docs: run the dev server with NODE_OPTIONS=--max-old-space-size=... and
   restart on RSS growth.

[silvertakana]

Independent repro on a different version and platform, same root object - plus the feed that sets the growth rate, three contributors your audit does not cover, and a caveat on stopgap #5.

I hit this on 0.1.7-rc.1, Windows 11, Node 25, but from heap snapshots rather than RSS. Line numbers below are from the shipped install (node_modules/@deepseek-ai/*); the modules are the same ones you audited in packages/.

Your item 3 confirmed, with the mechanism that sets the rate

The retention is the per-session in-memory log, and it is never unloaded - agreed. What determines how fast it fills is that every agent step appends a fresh full system-prompt render to it:

• dsh-agent-loop/lib/index.js:1032 const renderedPrompt = renderPrompt(assembly) -> :1041 this.session.append("system/message", { ... }) - every step, no memoisation.
• dsh-session/lib/index.js:1134 log = []; :1356 this.log.push(event) - no cap, no eviction, no trim; plus a second retained copy in this.eventsSnapshot ??= Object.freeze([...this.log]) (:1263), frozen and never rebuilt.

That also explains the deceleration you saw overnight: the rate is per-step, per-live-session, not per-wall-clock, so it is consistent with your "no new sessions opened" observation.

Measured growth, one process, two snapshots 34 minutes apart:

• heap selfSize 1,219,700,915 -> 2,184,664,310 B (+920 MiB); nodes +8,496,100; edges +25,235,268
• top growers: (object elements)/array +170.06 MiB, plain Object/object +66.55 MiB across +1,463,230 instances, system-prompt strings +29.34 MiB/155 instances
• normalised: ~4,300 retained objects per agent step (~715 objects/s sustained)

Four aborts in this log, all the same signature (Ineffective mark-compacts near heap limit / Reached heap limit): 3h01m at 4,029.8 MB against the then-4 GB cap, then 2h51m at 8,174.3 MB, 3h05m at 8,170.9 MB, and 3h54m at 8,163.0 MB after the cap was raised.

Three contributors beyond your list

1. Cold-corpus re-enumeration. dsh-session-persistence-jsonl list() walks every session directory and zstd-decompresses the first line of each header with no cache or TTL, and stat(id) embeds historicalCorpusRevision() (a full listGenerations pass plus sha256 per path), which makes a per-id call O(corpus). Callers that cannot pass a revision/since: dsh-session-query:283 and :399, dsh-session-query-sqlite:717 plus :730 (two enumerations per reconcile), dsh-workspace:891. Measured on a 1.5 GB / ~2,650-file corpus: 82 GB read and 31 GB written cumulative in under an hour of uptime - roughly one full corpus pass per minute.
2. Per-object amplification. dsh-util-values:46 walkJsonValue detaches every property with Object.defineProperty(...) (:53-58) as it walks, and calls hasIntrinsicConstructor (:14, a Function.prototype.toString.call) per object and array. Detaching properties drops objects into dictionary mode, so every retained event record costs more than its shape implies.
3. Third-party, in this install only: dsh-context@0.54.4 createToolAttribution (lib/index.js:2008) pushes a closure per wrapped tools service read into patched[] (:2041), drained only from its ctx.effect dispose (:2052-2054) - never, while the process lives. Measured 31,267 retained wrappedRegister closures from one file in a single process. Same family as your finding: nothing in the web path ever disposes. Drafted separately against that package's repo - happy to share it.

Caveat on stopgap #5: do not expect the bigger cap to buy time

Raising --max-old-space-size from 4 GB to 8 GB did not extend time to death here: 3h01m at 4.03 GB versus 2h51m and 3h05m at 8.17 GB. It does relocate the failure mode, though. The fourth incident reached a UI-frozen, constantly-compacting state at 8.16 GB without ever aborting - mark-compact spending 800+ ms per step, RSS thrashing - and needed a manual kill. So the useful half of your #5 is restart-on-growth, not the larger ceiling; a periodic restart is the only thing that has kept an unattended instance alive in this deployment.

Happy to post the full snapshot report (retainer walks, node-group anatomy, ownership attribution) if it would help - and agreed that #1 (an unload path that actually reaches the dropped AgentHandle.dispose(), so session/disposed -> retire() can fire) is the fix that generalises.

[longjucheng]

Reproduction on macOS at dsh-v0.2.0-rc.1.

Setup: macOS 15.7.5 (arm64), Node v22.23.1, source checkout at tag dsh-v0.2.0-rc.1, dev launch pnpm dsh web, default heap limit (~4.14 GB).

Crash: after 2.9 days of uptime, Ineffective mark-compacts near heap limit → abort through node::OOMErrorHandler. The crash report lists ~15.5k V8 memory regions (~3.9 GB of heap pages). The last mark-compact freed 5 MB in 2.9 s. The failing allocation was a string in node::fs::AfterScanDir. No tool call and no turn was open in any session at the crash; it occurred during a maintenance DarkWake.

Growth, from session logs (host-side time between step/end and the next step/start, same route throughout): p50 10 ms over the first two days → 38 ms → 0.07 s → 0.18 s → 0.41 s → 1.92 s → ~6 s per hour over the last six hours. Cheap tools (read/edit/grep) followed the same curve, 0.02 s to ~6 s p50.

At this version the per-step system-prompt append is deduplicated: SystemPromptProjection.project returns no commit when the rendered prompt equals the latest system message (packages/core/agent-loop/src/runtime-context.ts:88–102). The heap still reached the limit.

[VirtualHotBar]

Independent field reproduction on 0.2.0-rc.2 (latest stable), with growth numbers taken from systemd's own cgroup counters.

Environment: dsh 0.2.0-rc.2 (global npm install, dsh web --port 3080, headless on a Linux x64 server), Node v25.9.0, ~14 GiB RAM + 109 GiB swap, run under a systemd user unit with Restart=always (so the abort surfaces as an automatic restart rather than an outage).

Crash, 2026-10-09, instance age 1 d 20 h 33 m:

<--- Last few GCs --->
[1288:0x27b4b000] 152737440 ms: Scavenge (during sweeping) 1998.6 (2036.1) -> 1998.4 (2037.1) MB … allocation failure;
[1288:0x27b4b000] 152737687 ms: Mark-Compact 2259.7 (2297.6) -> 2016.2 (2112.7) MB … allocation failure; scavenge might not succeed
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

systemd recorded Main process exited, code=killed, status=6/ABRT and restarted it 6 s later.

One measurement that may matter for triage: the default cap on this host is 2.1875 GiB, not ~4.14 GB — require('v8').getHeapStatistics().heap_size_limit reports 2197504000 bytes under Node 25 here. So this class of deployment aborts after ~2 GB of live heap.

Growth, read out of systemd's per-unit counters (Consumed … memory peak / memory swap peak; they survive each stop, so no sampling is needed):

instance age	memory peak	swap peak	ended by
~20 h	3.5 GB	2.8 GB	host OOM-killer (SIGKILL)
~23 h	3.1 GB	2.3 GB	host OOM-killer
1 d 20 h	4.0 GB	3.2 GB	V8 heap abort (SIGABRT)
~4 d 21 h	14.4 GB	12.8 GB	manual restart

(The first two rows are a host-side incident of ours that is now fixed — the last two rows are the harness-side growth this thread is about, and the 14.4 GB instance has the same shape as the weeks-long case in the original report.)

Session volume here, for comparability with the "11 workspaces / 105 MB" figure in the OP: 3 workspaces, 834 MB of zstd JSONL across 1 185 session files, largest single session 38.9 MB compressed. Consistent with the per-opened-session accumulation model — this deployment is mostly idle overnight and still climbs.

We applied stopgap #5 verbatim (--max-old-space-size=6144 passed on the CLI, so only the main process is raised and the MCP children keep their defaults), plus a scheduled restart in a low-traffic window, and we can report back on how far the 6 GB cap carries us.

Question: has anything from the Ask list (web-session unload path / the dropped AgentHandle.dispose() chain, FrameQueue high-water mark, byte-bounded prepared-session cache) landed in 0.2.1-alpha.1 that would be worth testing against, or is "raise the cap + restart on a cadence" still the recommended posture for long-running dsh web deployments?