[Bug] Live sessions have no retention policy: heap grows with opened-session count and OOMs at any heap cap (128 GB host, measured per-session costs)

[503718696]

英文版（建议作为正文）

Environment

Item	Value
dsh	0.1.5-rc.1 (global npm install)
Node	v22.23.1
OS	Linux x64, 128 GB RAM (≈64 GB actually free at the time)
Launch	dsh web (web profile; 4 local profile plugins)
Heap cap	V8 default → 4144 MB; later runs with explicit --max-old-space-size

Summary

dsh web grows its heap in proportion to the number of sessions the process has opened, not merely to one large log, and nothing ever evicts those live sessions. Three separate runs died the same way at three different heap caps, and raising the cap only moved the wall:

run	heap cap	survived	crash signature
1	4144 MB (default)	19 h	Ineffective mark-compacts near heap limit
2	4144 MB (default)	16 h	same
3	8240 MB (--max-old-space-size=8192)	44 h	Reached heap limit

In every crash the last mark-compacts reclaimed almost nothing (e.g. 4030.2 → 4014.1 MB, i.e. 0.4 %), so the heap was live retained data, not uncollected garbage.

Measured: resident cost is per open session

Fresh boot with 8 sessions open (each holding an OS file lock on its session.lock), measured at the HTTP API (heapUsed ≈ 2875 MB):

session	events	decompressed log	resident heap	resident RSS
A	49,032	200.2 MB	708.4 MB	1195.9 MB
B	36,797	202.3 MB	679.9 MB	1168.7 MB
C	13,746	46.6 MB	176.4 MB	337.3 MB
D	11,061	40.1 MB	137.2 MB	289.5 MB
E	5,287	16.8 MB	63.2 MB	171.6 MB
F	1,310	6.6 MB	24.3 MB	95.3 MB
G	588	1.7 MB	9.5 MB	60.1 MB
H	384	1.5 MB	8.9 MB	58.2 MB
Σ	93,205	515.8 MB	1807.8 MB	3376.6 MB

The RSS sum (3376.6 MB) matches the process measurement (3354 MB) almost exactly; the heapUsed sum (1807.8 MB) accounts for ~63 % of the process's 2875 MB. Method: each transcript was streamed through the same path the host uses when it opens a session (line-wise JSON.parse → lossless clone → deepFreeze), one file per isolated process.

Measured: opening a session is one-way

I opened one not-yet-live session through the real wire protocol (/api/remote.mux, session/follow with maxMessages: 1), then closed the WebSocket. Evidence is the process's held session locks, which is not affected by background noise:

moment	sessions held by the process	heap
before	8	3044 MB
6 s after open	9	3015 MB
3 s after socket close	9	3015 MB
20 s after socket close	9	3029 MB

The count never returns. Repeating page loads/streams accumulates.

Measured: growth does not require activity

A /proc-based sampler watched one process for its whole life (3,861 samples): anonymous memory went 1.05 GB → 4.64 GB in 16.2 h, monotonic. During that window there were zero session-log writes and zero queue/storage writes — only a handful of attachment files. So the slope is not "work done"; there is an additional time/aggregate-proportional term.

Upstream source locations (checked against master, 0.1.6-alpha.1, pushed 2026-09-15 — all still present)

Verified by fetching the files and grepping the current master tree:

1. No live-session retention policy.
   packages/api/session-controller/src/client/sessions/manager.ts

   • :96 private readonly sessions = new Map<SessionId, Session>()
   • :292 this.sessions.set(sessionId, session) (lazy build, never evicted)
   • :245 async drop(sessionId) exists; :247 this.sessions.delete(sessionId)
   • grep -nE "idleTimeout|maxSessions|evict|retention" over the package: no hits

2. A read handle pins the whole decoded log.
   packages/session/session-persistence-jsonl/src/index.ts

   • primed — 2 occurrences on master
   • COLD_LOG_MEMO — 2 occurrences (the decoded-log memo, count-bounded by entries, not bytes)
   • decodeStoredLog 3, freezeStoredEvents 3, memoizeStoredLog 4

3. The observation cache can exceed its own cap.
   packages/session-query/session-query/src/index.ts:122-131

   • const preparedSessionCacheSize = config.preparedSessionCacheSize (default 5)
   • entries pinned by an active lease (refs > 0) are skipped by eviction, so a cache bounded by count is not a bound in bytes.

4. The raw log is append-only and compaction does not trim it. Session.log keeps every streaming chunk; surface replace removes shadowed entries from model input only.

Items 1–4 are source reading + sampling. I do not have a heap snapshot from the crashing process (it aborted with SIGABRT and no .heapsnapshot was written), so “which single holder dominates” is an analysis, not a measurement. Independent reports (#3275, #5995, #5696, #6511, #4722) reach the same conclusion with heap snapshots of their own.

Why "more RAM" is not the fix

The crash is not physical-memory exhaustion: a 64 GB host still died at ~4.1 GB (#5995), and one reporter measured that with --max-old-space-size=16384 the heap still climbs past 4 GB with no plateau (#3275). The resident term is opened sessions × per-session cost, so any fixed cap is eventually crossed.

Reproduction (minimal)

# 1. a session store with many/large sessions
ls -la "$DSH_HOME/sessions"/*/ | head

# 2. start with a known cap and no client
NODE_OPTIONS=--max-old-space-size=8192 dsh web --no-open

# 3. sample heap every 60 s (no client needed) — RSS/heap climbs monotonically
while :; do grep VmRSS /proc/$(pgrep -f 'dsh/lib/bin.js web')/status; sleep 60; done

# 4. connect one client and open a *stored* session; then close the socket.
#    The server keeps the session resident: watch the held session locks
ls -l /proc/$(pgrep -f 'dsh/lib/bin.js web')/fd | grep -c session.lock

What I ruled out on my host

• Browser side: the server paginates the opening frame (DEFAULT_MAX_MESSAGES = 50), and the leak reproduces with no client at all (--no-open), so the renderer is not the cause of the host heap.
• Legacy session format: bcahlit in Web server OOMs within ~60s of any client connecting when a workspace contains a large session #3275 showed hiding all v2 files and keeping only v3 still OOMs.
• Projection cache: same reporter — moving session_projcache aside makes it OOM faster.
• Third-party plugins: disabling a 21-row community bundle cut the slope ~4× (75 → 17 MB/s) but did not stop it.
• Invalid agent preset ([dsh] 非法 preset 配置 → cordis 无限 reload + ~2GB 内存泄漏（~20 分钟后 OOM），且完全静默 #6415's trigger): checked — my own preset has the required persona.prefix, so that path is not active here.
• Not corruption: no SessionPersistenceCorruptionError / SessionFormatUnsupportedError in ~10 boots; only heap exhaustion. All sessions load.

Suggested fixes (ordered by leverage)

1. Give live Sessions/Agents a retention policy: idle timeout and/or a byte-aware cap with real eviction (manager.get() ↔ drop()), plus a public eviction entry point — today the only drop() caller is scope teardown.
2. Don't retain the decoded log in the handle state (primed) for read handles; stream the fold instead of materialising every event.
3. Make both memos byte-bounded (they are count-bounded today: 2 entries and 5 entries) and let leases expire.
4. Bound the resident reconstruction window separately from durable history (see the community handbook's four-budget model).

Related

• Web server OOMs within ~60s of any client connecting when a workspace contains a large session #3275 — Web server OOMs within ~60s of a client connecting when a workspace contains a large session
• V8 heap OOM after ~8h: opened Conversations are fully hydrated and never evicted (no live-session retention policy) #5696 — V8 heap OOM after ~8h: opened Conversations fully hydrated and never evicted
• harness 进程内存随会话事件数持续增长，长批次约 8 小时后堆触顶 OOM 退出（附测量与采样归因） #5995 — 内存随会话事件数持续增长，约 8 小时后堆触顶（含 26.1 MB/会话 的官方基准引用）
• [Bug] Every session open / client reconnect re-decodes the session log and retains the decoded copy until V8 heap OOM (~30–90s of UI activity) #6511 — every session open / reconnect re-decodes the log and retains the decoded copy
• [bug] In-memory session log is unbounded (append-only) and compaction never trims it - long/heavy sessions OOM the web process #4722 — in-memory session log is unbounded (append-only), compaction never trims it
• [bug反馈] 超长会话因流式增量碎片（reasoning-chunks / text-chunks）未自动清理导致内存溢出 (OOM) 与历史加载失败 #5347 — 超长会话因流式增量碎片未清理导致 OOM

---

中文版（可放在英文之后，或作为主帖）

环境

@deepseek-ai/dsh 0.1.5-rc.1（全局 npm 安装）· Node v22.23.1 · Linux x64 · 物理内存 128 GB（实际可用约 64 GB）· web profile（含 4 个本地 profile 插件）· 启动方式 dsh web。

结论

dsh web 的堆占用与**「该进程打开过的会话数量」成正比（不只是单个大日志），而且没有任何驱逐机制**。三次运行在三个不同的堆上限上以同一方式崩溃，抬高上限只是把墙往后挪：

运行	堆上限	存活	崩溃签名
1	4144 MB（默认）	19 小时	Ineffective mark-compacts near heap limit
2	4144 MB（默认）	16 小时	同上
3	8240 MB（--max-old-space-size=8192）	44 小时	Reached heap limit

每次崩溃前最后一次 mark-compact 几乎回收不到东西（例如 4030.2 → 4014.1 MB，即 0.4%），说明堆里是活着的保留数据，不是没回收的垃圾。

实测一：常驻成本按「每个已打开会话」计

全新启动、8 个会话处于打开状态（各自持有 session.lock 的 OS 文件锁），经 HTTP API 读到 heapUsed ≈ 2875 MB：

会话	事件数	解压日志	常驻 heap	常驻 RSS
A	49,032	200.2 MB	708.4 MB	1195.9 MB
B	36,797	202.3 MB	679.9 MB	1168.7 MB
C	13,746	46.6 MB	176.4 MB	337.3 MB
D	11,061	40.1 MB	137.2 MB	289.5 MB
E	5,287	16.8 MB	63.2 MB	171.6 MB
F	1,310	6.6 MB	24.3 MB	95.3 MB
G	588	1.7 MB	9.5 MB	60.1 MB
H	384	1.5 MB	8.9 MB	58.2 MB
合计	93,205	515.8 MB	1807.8 MB	3376.6 MB

RSS 合计（3376.6 MB）与进程实测（3354 MB）几乎完全吻合；heap 合计占进程 2875 MB 的约 63%。测法：把每份日志分别喂给一个独立进程，走 host 打开会话时的同一条路径（逐行 JSON.parse → 无损克隆 → deepFreeze）。

实测二：打开会话是单向的

我用真实线协议（/api/remote.mux + session/follow，maxMessages: 1）打开一个当时尚未 live 的会话，然后关闭 WebSocket。判据用进程持有的会话锁数——它不受后台噪声影响：

时刻	进程持有会话数	heap
打开前	8	3044 MB
打开后 6 秒	9	3015 MB
关闭 socket 后 3 秒	9	3015 MB
关闭 socket 后 20 秒	9	3029 MB

计数不回落。 反复刷新页面/重连会持续累加。

实测三：增长不需要活动

用一个只读采样器盯住某次进程的完整生命周期（3,861 个样本）：匿名内存 16.2 小时内从 1.05 GB 单调涨到 4.64 GB。该时间窗内会话日志零写入、队列与 storages 零写入（只有几个附件文件）。所以斜率不是「做了多少活」，另有一项与时间/累计量成正比的增长。

上游源码位置（已对照 master / 0.1.6-alpha.1，最后推送 2026-09-15，四处都还在）

1. live 会话没有保留策略：packages/api/session-controller/src/client/sessions/manager.ts
   • :96 private readonly sessions = new Map<SessionId, Session>()
   • :292 this.sessions.set(sessionId, session)（懒建，永不淘汰）
   • :245 async drop(sessionId) 存在、:247 this.sessions.delete(sessionId)
   • 对 idleTimeout|maxSessions|evict|retention 全包 grep：零命中
2. 读句柄钉住整份解码日志：packages/session/session-persistence-jsonl/src/index.ts —— primed 2 处、COLD_LOG_MEMO 2 处、decodeStoredLog 3 处、freezeStoredEvents 3 处
3. 观察缓存可超出自身上限：packages/session-query/session-query/src/index.ts:122-131，preparedSessionCacheSize 默认 5，且 refs > 0（被租约钉住）的条目在淘汰时被跳过 —— 一个「按条数」的上限在字节维度上不构成上限
4. 原始日志 append-only 且 compaction 不裁剪：Session.log 保留全部流式碎片；surface replace 只影响模型输入，不删原始事件

上述 1–4 是源码阅读 + 采样归因。我没有拿到崩溃进程的 heap snapshot（它以 SIGABRT 退出、未落盘 .heapsnapshot），所以「单一主导持有者」属分析判断而非定论。#3275 / #5995 / #5696 / #6511 / #4722 等报告各自带堆快照，得出相同结论。

为什么「加内存」不是解法

这不是物理内存耗尽：一台 64 GB 的机器同样死在 ~4.1 GB（#5995）；另有人实测 --max-old-space-size=16384 时堆仍越过 4 GB 且无平台期（#3275）。常驻项是 已打开会话数 × 单会话成本，因此任何固定上限终将被越过。

已排除项（避免后来者重复踩）

• 不是浏览器侧：开局帧被分页（DEFAULT_MAX_MESSAGES = 50），且完全没有客户端（--no-open）时同样复现。
• 不是旧格式：隐藏全部 v2、只留 v3 仍 OOM（Web server OOMs within ~60s of any client connecting when a workspace contains a large session #3275 评论）。
• 不是投影缓存：把 session_projcache 移开反而更快 OOM（同上）。
• 不是第三方插件：关掉 21 行社区 bundle 使斜率降约 4×（75 → 17 MB/s），但没有停止。
• 不是非法 preset（[dsh] 非法 preset 配置 → cordis 无限 reload + ~2GB 内存泄漏（~20 分钟后 OOM），且完全静默 #6415 的触发点）：已自查，本机 preset 的 persona.prefix 存在，该路径未激活。
• 不是数据损坏：约 10 次启动中没有任何 SessionPersistenceCorruptionError / SessionFormatUnsupportedError，只有堆耗尽；全部会话可加载。

建议修法（按杠杆排序）

1. 给 live Session/Agent 加保留策略：空闲超时和/或字节感知的上限 + 真正的驱逐（manager.get() ↔ drop()），并补一个公开驱逐入口——今天 drop() 的唯一调用者是 scope teardown。
2. 读句柄不再保留解码日志（primed）；把 fold 改成流式，而不是物化每个事件。
3. 两个 memo 改成按字节封顶（现在是按条数：2 条与 5 条），并让租约到期。
4. 把「常驻重建窗口」与「持久历史」分开设界（参见社区手册的四预算模型）。

相关讨论

#3275 · #5696 · #5995 · #6511 · #4722 · #5347

[PerryLink]

Your finding is confirmed on the static-source side at HEAD (ddefc45fbc, v0.1.6-alpha.2), and there's a line in the source that states the behavior is deliberate.

packages/api/session-controller/src/client/sessions/manager.ts:92:

private readonly sessions = new Map<SessionId, Session>()

Lazy-built and inserted in get() at :239-243, with no removal path other than drop() (:193-199). drop()'s only callers are service.ts:601 (inside retireScope) and :700 (the startScopeDrop default argument); retireScope itself only runs when the reference count hits zero (service.ts:570) or a scope fiber is torn down (:624). I grepped the entire repo for idleTimeout|maxSessions|sessionRetention|evictSession — zero hits. There is no time- or size-based eviction anywhere.

The design intent is written down in packages/api/session-controller/src/client/sessions/session.ts:1:

// Sessions remain resident after creation so their open Remote sources keep running off-screen.

So "open a session and it stays" is current design, not a regression — and it's consistent with your measurement that the held-session count never returns after socket close.

On config knobs: the only adjacent one is preparedSessionCacheSize (packages/session-query/session-query/src/config.ts:25, default 5, consumed at index.ts:122-131), but it bounds a different cache — the cold observation cache — and pinned entries are exempt (observation.ts:65: "a pinned entry (refs > 0) is never evicted"). It cannot bound live sessions. --max-old-space-size is a Node flag, not a harness knob; your three-run data showing it only moves the wall is consistent with retained-live-data growth. Your PAGE_MESSAGES = 50 claim for the opening frame also checks out (session.ts:51).

Practical mitigation today: nothing in source reclaims live sessions, so a scheduled restart of dsh web is the only reliable way to reclaim that heap. Your measured tables, RSS sums, and the "which holder dominates" attribution are your measurements (you flagged the missing heap snapshot yourself) — I confirmed the source facts they rest on.

If you still have the per-session numbers from run 3's end state, a before/after-restart heap comparison would make the "restart reclaims" case airtight.

[503718696]

English (post this)

Thanks — confirming the source facts at HEAD plus the design-intent comment is exactly what I could not do from a packaged install, and the three line-number sets now on record are themselves informative: your manager.ts:92/:239-243/:193-199 (HEAD ddefc45fbc, v0.1.6-alpha.2), the master numbers I reported, and the ones in the built 0.1.5-rc.1 tree all differ. Anyone anchoring a patch on those coordinates will be re-anchoring on every release.

Here is the restart comparison you asked for. I did not capture the old process's final heap (its --inspect sampler had already stopped), so I mark that one cell as unmeasured rather than inventing a number.

Run 3 end state → restart

Observation	Value	How
Run 3, last sampler reading (uptime 9 h 20 m)	5,177 MB heapUsed	inspector Runtime.getHeapUsage, 60 s cadence
Run 3 peak (it was terminated by me, not by V8)	≤ 8,240 MB	it never reached the cap — no OOM/abort in its log
Run 3 sessions held at that point	8–10	ls -l /proc/<pid>/fd | grep -c session.lock
Fresh boot of the restarted host (~5 min in)	2,874 MB heapUsed	same inspector method
Sessions held by the fresh host	8–10	same fd method
Reclaimed by restart	≈ 2.3 GB	5,177 − 2,874, same session set (to within one)

The one-session difference matters, and it works against the claim rather than for it: the fresh host still held an extra session that was open at the time of the comparison, so the figure is if anything conservative.

The part that makes it airtight (new data, 38.5 h later)

The same host, restarted with an explicit 32 GB cap, has now been up 38.5 hours:

Measurement	Value
heapUsed / heapTotal	7,200 MB / 7,425 MB
process RSS / anonymous	7,853 MB / 7,785 MB
sessions held	10–16

This is the decisive comparison, because memory in each process tracks held sessions, not the previous process's residue:

Host	cap	uptime	sessions held	heapUsed
old (default-ish cap 8,240 MB)	8,240 MB	9 h 20 m	8–10	5,177 MB
new (32 GB cap)	32,816 MB	38.5 h	10–16	7,200 MB

If a restart did not reclaim, the new process would have started from the old retained set and then grown for another 38.5 h — it would be far past 7.2 GB and past the old cap. It instead started ~2.3 GB lower than where the old one died, and its current footprint is explained by the sessions it holds (~0.4–0.5 GB per session on this host), which is consistent with the per-session table in the opening post (8.9 MB for a 384-event session up to 708 MB for a 49,032-event session).

So: a scheduled restart reclaims the heap, and since nothing in source reclaims live sessions, that is currently the only reliable mitigation — as you said.

Two clarifications on the numbers you cited

1. On preparedSessionCacheSize: agreed it bounds a different cache, and the pinned-entry exemption is the same in the shipped build I measured (evictPastCapacity skips candidate.refs > 0). My report cited it only as a secondary bound that does not hold in bytes, not as the live-session bound.
2. On the repo's own gate figure (Agent retained heap = 26.1 MB, CI budget 33 MB): my measurements are much larger in every real case — 8.9 MB for the smallest session I measured, then 63 MB, 137 MB, 176 MB, and 679–708 MB for the two large ones. I read that as "26.1 MB is a starter-session best case", while the resident term scales with event volume per session, so the number to budget against is the per-session log, not a fixed per-session constant. If the maintainers read that differently I would like to be corrected.

Reproduction for anyone at HEAD

No client is needed and nothing has to be modified:

# 1. capture the post-boot baseline
node --inspect=127.0.0.1:9229 <path-to>/dsh/lib/bin.js web --no-open &
node -e 'fetch("http://127.0.0.1:9229/json/list").then(r=>r.json()).then(async l=>{
  const ws=new WebSocket(l[0].webSocketDebuggerUrl);
  ws.onopen=()=>ws.send(JSON.stringify({id:1,method:"Runtime.getHeapUsage"}));
  ws.onmessage=e=>{const m=JSON.parse(e.data);if(m.result)console.log((m.result.usedSize/1048576).toFixed(0)+" MB",(m.result.totalSize/1048576).toFixed(0)+" MB");};
})'

# 2. open one stored session over the real wire protocol, then close the socket,
#    and watch the held-session count (it never returns)
ls -l /proc/$(pgrep -f 'dsh/lib/bin.js web')/fd | grep -c session.lock

# 3. restart, re-read step 1, and compare against the held-session count

I kept a sampler running on this host (60 s cadence: heapUsed / heapTotal / RSS / anonymous / held-session count) and can post the full multi-day curve — plus the per-session cost table as CSV, with session ids and timestamps stripped — if that is useful as input to a regression gate.

---

中文版（可作补充）

感谢确认——能拿到 HEAD 的源级事实和那行「设计意图」注释，是我从打包安装里做不到的。你给的行号（HEAD ddefc45fbc / v0.1.6-alpha.2 的 manager.ts:92/:239-243/:193-199）、我报的 master 行号、以及我本地打包树 0.1.5-rc.1 的行号三者都不一样——这本身就说明：任何按行号锚定的补丁，每次发版都要重新锚定。

重启对比数据如下。我没有测到旧进程终止前一刻的堆（它的采样器已提前停止），所以那一格标为未测，而不是编一个数。

观测	数值	方法
旧进程最后一次采样（存活 9h20m）	5,177 MB heapUsed	inspector，60 秒一次
旧进程峰值	≤ 8,240 MB	它是被手动终止的，从未触顶、日志无 OOM
旧进程当时持有会话	8–10	/proc/<pid>/fd 计数
新进程启动后约 5 分钟	2,874 MB heapUsed	同上
新进程当时持有会话	8–10	同上
重启回收量	≈ 2.3 GB	5,177 − 2,874，会话集相同（相差不超过 1 个）

那 1 个会话的差异是对结论不利的方向：新进程当时仍多持有 1 个会话，所以这个对比偏保守。

更硬的一条（38.5 小时后新增数据）：同一台机器重启并显式设 32G 上限后已跑 38.5 小时：heapUsed 7,200 MB、RSS 7,853 MB、持有 10–16 个会话。对照表：

进程	上限	存活	持有会话	heapUsed
旧	8,240 MB	9h20m	8–10	5,177 MB
新	32,816 MB	38.5h	10–16	7,200 MB

如果重启没有回收，新进程会从旧进程的残留集起步再涨 38.5 小时——它早就越过 7.2GB、也越过旧上限了。事实相反：它比旧进程死亡处低了约 2.3GB 起步，而当前占用可由它持有的会话数解释（本机约 0.4–0.5 GB/会话），与首帖的逐会话表一致（384 事件 8.9MB → 49,032 事件 708MB）。

结论：定期重启是目前唯一可靠的缓解手段，因为源码里没有任何东西回收 live 会话——这与你的判断一致。

两点澄清：① preparedSessionCacheSize 我同意它限的是另一个缓存，pinned 豁免（refs > 0 不淘汰）在我实测的打包版本里同样存在，我报告里只把它当作「按字节不成立」的次要界，不是 live 会话的界。② 仓库自身基准的 26.1 MB/会话 与我的实测差距很大（最小 8.9MB → 最大 708MB），我的读法是：26.1MB 是起步会话的最好情况，而常驻项随每会话事件量增长，所以该拿去预算的是每会话日志规模，而不是一个固定的每会话常数。若维护者另有读法，欢迎纠正。

复现步骤（HEAD，无需客户端、无需改代码）见英文版。本机采样器仍在跑，需要的话我可以补上完整的多日曲线，以及去掉会话 id 与时间戳的逐会话成本 CSV，供回归门禁使用。