[bug] In-memory session log is unbounded (append-only) and compaction never trims it - long/heavy sessions OOM the web process

[Shirlif]

Summary

The dsh web server (Node, apps/cli/lib/bin.js web) crashes with
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
after sustained use in a long / heavily-orchestrated session. The process reaches ~4 GB of live V8 objects and aborts, taking all in-flight background work (subagents) with it.

Root cause is an unbounded in-memory session event log that compaction never trims, compounded by continuable subagents that stay resident during a wide fan-out.

Environment

• Node 22.22.2 (also reproduced under Node 22.x generally), Windows 10/11, 16 GB RAM.
• node apps/cli/lib/bin.js web (the web host). --max-old-space-size is not set anywhere; the crash is at V8's default heap ceiling (~4 GB on this host).
• Reproduced with the agent preset that composes both subagent and subagent_fork with backgroundMode: continuable (heavy multi-agent orchestration).

Reproduction / trigger

Run one long-lived session that either:

1. accumulates a large conversation (many turns, large tool results, images/docs), and/or
2. performs a wide parallel fan-out of subagents (many subagent / subagent_fork calls in a single wave, each returning a sizable report into the parent session).

The in-memory heap grows monotonically over the session; the model-facing context may appear bounded (because compaction is on) while the process's live objects keep climbing to the heap cap and then abort.

Root cause

1. Session.log is append-only and never pruned, with no length/byte cap.
packages/core/session/src/index.ts:426 declares private log: SessionEvent[] = []; append() only pushes (index.ts:643). There is no session-length limit, event-count cap, or history eviction anywhere in packages/core/session (the log is the durable "session is the source of truth" record, retained for replay/resume).

2. Compaction reduces only the model surface, never the log — so it does not free the heap.
A compaction replace surface op does state.nodes.splice(...) (packages/core/session/src/surface.ts:368-371), removing seqs only from the surface node list. It never removes events from Session.log. So even with automatic compaction firing at thresholds, heap grows with the conversation; the model context is bounded but the retained event log is not.

3. Continuable subagents stay resident until they settle.
SubagentContinuationManager.activations = new Map<SessionId, Activation>() (packages/subagent/subagent/src/continuation.ts:357), where each Activation holds an AgentHandle with its own full Session (full log). With the tools configured backgroundMode: continuable, a normal call returns a subagentId immediately and the child stays resident in memory until it settles. During a wide parallel fan-out the whole tree is resident simultaneously. Settled children are eventually disposed (and their sessions freed), but every settling child's output is also delivered back to the parent as a subagent-settled notice and a tool-subagent-report, both appending content to the parent session log — so the parent log is the genuinely unbounded term.

Impact

• The dsh web process aborts; any in-flight background subagents / jobs in the same process are lost (their running turns are not replayed after restart).
• The parent session's on-disk log keeps growing; a higher --max-old-space-size merely delays the crash and can push the host into swap.

Proposed direction (design discussion welcome)

The durable fix is to bound the in-memory session log. Two candidate directions:

• A (contained): compact-and-trim. When a compaction/replace commits, also drop the superseded events from the in-memory Session.log (older history becomes the compact summary). Memory becomes bounded. Trade-off: the GUI/row history no longer shows the pre-compaction turns in full (they exist only as the summary), unless a cold on-disk copy is kept for lazy rendering.
• B (thorough): bounded in-memory window over a disk-backed log. Keep only a recent window + compaction summary in memory; the full event log already lives on disk (.jsonl.zstd) and old history is lazy-loaded for the GUI/replay. Memory bounded and full history preserved. Larger change: replay, subagent, lifecycle, and GUI render paths must handle the window.

Both must respect the repo invariant "Model-visible ⟺ logged" (anything reaching a model request is reconstructable from the log) — after a compaction the summary is what the model sees going forward, so trimming the superseded events stays consistent. A config knob for the bound should be a validated Config field (repo rule: no hardcoded tunables).

Separately, making completed continuable subagents dispose sooner (and evict idle ones) reduces the fan-out spike, but does not fix the unbounded parent log.

Notes / evidence sources

• packages/core/session/src/index.ts (log: SessionEvent[], append, SessionStore.store Map)
• packages/core/session/src/surface.ts (SurfaceManager.applySurfacePlan splice; README: "The append-only log remains the source of truth")
• packages/subagent/subagent/src/continuation.ts (activations, watchSettlement, finishDisposal)
• packages/session/session-persistence/README.md: "No deletion or retention API — pruning stored sessions is out-of-band backend maintenance" (same for the projection cache) — i.e. there is currently no built-in mechanism to bound this.
• Reproduced while using the official subagent / subagent_fork tools configured backgroundMode: continuable.

Happy to provide additional detail, a minimal reproduction, or a draft patch under one of the proposed directions.

[weijiafu14]

Your two causes are separable and both look right to me. I can add a third holder of the same bytes, a candidate invariant, and two sibling threads that are the same session shape seen from other angles.

1. The accumulated chunk text is held more than once, and plugins are one of the holders.

My plugin subscribes to session/event and, on every assistant/chunk, accumulates text so it can project a streaming-update event for the extensions running on top of it. That is a second in-memory copy of the same deltas, in the same process, for as long as it is kept.

The thing that keeps ours bounded is worth stating because it is exactly the rule the store is missing: we key the accumulator by session:turn:step and delete the entry when the message closes. Once the terminal assistant/message arrives, the accumulated string is redundant with it, so we drop it.

2. That is a candidate invariant for the store, and another thread has already measured its value.

#4678 measured a 50-message history page: 38,483 events / 9.0 MB, of which 99.4% were assistant/chunk deltas belonging to messages that had already closed, and shows the terminal assistant/message carries the complete final block list. That is the read-path proof of the same redundancy you are hitting on the memory path. Stated as one rule for both:

Once a message closes, its chunk deltas carry no information the terminal assistant/message does not.

#4678 applies it on read. Your report is what happens when it is never applied in the store at all.

3. A third sibling, which sets a lower bound on the sizes involved.

#4633 reports one assistant/message carrying 254,780 sourceEventSeqs — enough to crash paginate outright. That is a quarter-million chunk events absorbed into a single message, in one session, in the wild. It is direct evidence for the scale you are describing, and it is worth citing in your report: your OOM is not an exotic workload, it is what that array size implies about the log behind it.

4. On your second cause (resident continuable subagents). Worth keeping separate from cause 1 in the report, because it has a different owner and a different fix. Two adjacent threads:

• Sub-agent completion polling fails with "unknown job" during multi-agent task, halting generation mid-task #4696: a parent that ended its turn to wait for a subagent settlement notice, and was never woken — a stalled continuable child, still resident.
• [Feature Request]Goal rounds busy-loop while long-running subagents are still active #4715 / [Bug]？ goal 功能开启后 agent 无法被动等待后台任务：每次进入 idle 即被强制注入 <goal_round>，空转消耗轮次与 token #4664: the opposite failure on the same seam, where an idle parent is re-armed on every idle tick while work is outstanding.

Both are about lifetime of outstanding children, which is where "stay resident during a wide fan-out" ultimately gets decided. If cause 1 is fixed and cause 2 is not, a wide fan-out will still hold every child's session in memory — just with less log per child.

What I cannot tell you: whether trimming closed-message chunks in the store is safe for every consumer. It is safe for us only because we consume the live stream and drop our copy at message close — a consumer that cold-resumes and rebuilds in-progress state from the store would see terminal messages only. That is correct final content but a different object. Worth naming as a compatibility question in the proposal rather than discovering it later; I raised the same seam on #4678.

Interest disclosure: the plugin above is mine. The session store, compaction and the subagent runtime are DSH's own components — we do not touch them and could not fix any of this. The only first-hand thing here is that we hold a second copy of these bytes and how we bound it.

[denial123789]

The rc.2 source supports the budget separation in this report:

• Session owns private log: SessionEvent[] = [], and append() pushes every accepted event.
• a surface replace splices state.nodes; the Session contract explicitly says this removes entries from future model input without deleting raw log records.
• SessionStore retains live entries by scope, while the continuation manager retains an AgentHandle per activation until settlement and child-first disposal.

That proves an unbounded retained term exists for a continuously appended live Session. It does not yet prove the raw array is the only dominant holder or that ~4 GB is portable; a heap dominator view plus a fixed-interval series of heapUsed, RSS, events-by-type/bytes, live Sessions, and activation states would close that attribution.

I would be cautious with candidate A if “trim” means changing the durable log. The current contract also uses older events for human transcript semantics, sourceEventSeqs validation, fork boundaries, usage/audit/plugin projections, and cold replay. A safer compatibility direction is three explicit tiers:

1. complete append-only durable archive;
2. byte-bounded resident reconstruction window;
3. independently bounded/rebuildable projection caches.

Closed-message chunk payload reclamation looks promising, but needs tests for in-progress crash repair, chunk-consuming plugins, usage chunks, source-reference resolution, and identical cold-derived final messages. Event count alone is not a sufficient bound because payload sizes differ radically.

For reproduction, I would separate four cases: event volume without fan-out, fan-out with a short parent, the exact first compaction crossing, and closed-chunk density. That distinguishes a cumulative parent term from a temporary resident child-tree peak. For operations, a soft threshold should stop new admission and flush; a hard threshold should cancel, reject new model/tool admission, flush within a deadline, and exit with a typed reason. Raising --max-old-space-size only moves the boundary.

I maintain the independent DeepSeek Harness Handbook. I wrote the full source-pinned operator map, diagnostic bundle, failure router, and 20 regression gates here: https://sandbaseai.github.io/deepseek-harness-handbook/session-heap-growth.html