Repository navigation
绕开会话历史 OOM:历史载荷留在磁盘、按需回读 | Working around the session-history OOM #13334
NothingToSay0031
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
问题
不少人都遇到过:DSH 宿主会把整份会话日志解析进内存,所有事件全量常驻。工具结果(
tool/result)单条可达数 MB,长会话(几十万事件)的载荷能占到几个 GB。桌面宿主跑在 Electron 的 Node 模式下,V8 堆上限固定约 4192 MB,外部无法调高,于是打开或累积这类会话最终以FATAL ERROR: JavaScript heap out of memory(exit 134)收场。已有的报告从不同角度描述了同一个形状:
对我有效的绕法
我写了一个插件 dsh-session-reaper,针对「载荷全量常驻」这一环:
https://github.com/NothingToSay0031/dsh-plugins/tree/master/dsh-session-reaper
lazyOpen默认开启,包装 jsonl 后端的解码入口,对不小于阈值(默认 16 MB)的日志在打开时返回冻结的延迟视图,而不是整份解析常驻。视图的身份字段(type/seq/time)是真实值,载荷变成 getter,第一次被读到时才从磁盘上原始的 zstd 日志解压读回,读过的随即经有界 LRU 驱逐。会话构造期的校验与折叠仍会逐条读一遍载荷,但常驻内存由窗口决定,不随日志大小增长。回读用宿主自己的权威事件词汇集做准入校验;词汇集、帧索引或头部不变量无法证明时,整条懒路径回退原始解码。MIT 许可,带测试,还有 dry-run 模式可以先测量再启用。
诚实的边界
希望官方做的
持久的修复应该在核心层,方向与已有帖子的诉求一致:
欢迎试用并反馈什么有效、什么无效;实现细节和测量数据也可以提供。
English version
The problem
Many of us have hit this: the DSH host parses the entire session log into memory and keeps every event resident. Tool results can be megabytes each, so a long-running session (hundreds of thousands of events) can hold gigabytes of payload. The desktop host runs in Electron's Node mode with a fixed ~4192 MB V8 heap cap that cannot be raised from outside, so opening or accumulating such sessions ends in
FATAL ERROR: JavaScript heap out of memory(exit 134).Existing reports describe the same shape from different angles:
A workaround that works for me
I wrote a plugin, dsh-session-reaper, that attacks the resident-payload part of the problem:
https://github.com/NothingToSay0031/dsh-plugins/tree/master/dsh-session-reaper
lazyOpen(on by default) wraps the jsonl backend's decode entry point and returns frozen lazy views for logs at or above a threshold (16 MB by default) instead of a fully parsed resident array. Identity fields (type/seq/time) stay real; the payload becomes a getter that decompresses from the original on-disk zstd log on first access, and what gets read is evicted through a bounded LRU right away. The session constructor still validates and folds every event (reading each payload once), but residency is set by the window, not by the log size. Readback is admitted against the host's own authoritative event vocabulary; whenever that vocabulary, the frame index, or the header invariants cannot be proven, the whole lazy path falls back to eager decoding.MIT-licensed, with tests and a dry-run mode for measuring before enabling.
Honest limits
What I'd love to see upstream
The durable fix belongs in the core, along the lines the existing threads already ask for:
Happy to share implementation details or measurements if useful. If you try the plugin, reports of what worked / what didn't are very welcome.
All reactions