persistence: one corrupt session file crash-loops the whole host process — and the zstd frame layout that triggers it is undocumented

[bluewowo]

Summary

If any single session file under $DSH_HOME/sessions/** is in a shape the reader rejects, sessionPersistence.list() throws corrupt Zstandard session log: … during workspace-registry initialization, and the entire host process enters a crash loop (we observed systemd NRestarts=13, HTTP 000). The Web UI then shows a "fresh" onboarding state — all history appears gone. One bad file in one workspace takes down the service for all workspaces.

We hit this twice in one day (2026-08-16/17) while repairing poisoned session records by hand — which leads to the second half of the post: the on-disk frame layout is nowhere documented, so any operator editing a session file will break it.

What we did wrong (so you can see how easily this happens)

A session log (session.jsonl.zstd) is multi-frame zstd:

• frame 1 must be exactly one line — the {"type":"session", …} header
• subsequent frames hold complete JSONL records (the writer flushes in batches)
• a torn final frame is tolerated (crash-tail repair)

Our repair script re-compressed the file as one frame containing all lines. The reader then rejects it with:

Error: corrupt Zstandard session log: first frame is not exactly one header line

Nothing in the docs (including docs/persistence-catalog.md) describes this layout. The only way to learn the contract is to read the reader's source (assertZstdHeaderFrame et al.). We learned it the expensive way.

The two incidents

1. Host crash loop (severe): one single-frame file in workspace --home-jojo-dsh-- → sessionPersistence.list() threw at registry init → the web instance (rc.5 checkout under systemd, Restart=always) crash-looped (NRestarts=13, HTTP 000) → the Web UI showed a blank/onboarding state and every session looked "lost". Fixed by re-framing the file (header frame + one record per frame, ~2900 frames, content byte-identical) → service recovered, all history intact.
2. Silent per-session failure (moderate): the same wrong framing on a bot-channel session made agents.resume throw for that session only; the channel plugin's error path swallowed it, so inbound messages were dropped without any reply or log. (The swallow itself is reported against the plugin separately.)

What we'd like to discuss

1. Blast radius: should list() isolate a corrupt session (skip + log loudly + optionally quarantine the file, e.g. rename to *.corrupt-<ts>) instead of failing the whole registry init? A corrupt session can still be surfaced as a per-session error when someone actually opens it. As-is, one bad file is a total outage of the host, and under Restart=always it's a crash loop with no progress.
2. Documentation: the zstd frame layout (header frame = exactly one line; record frames; torn-tail tolerance) is a load-bearing contract for anyone doing operational repairs (and we did need to do them). A section in docs/persistence-catalog.md — plus a short "manually repairing a session file" note (re-frame as header frame + one record per frame; keep content byte-identical) — would save the next operator an hour.
3. A validator (nice to have): a dsh doctor-style check that scans all session files and reports corrupt ones (with the specific frame/line) would turn "why is my service crash-looping / this session unreadable?" into a one-liner.

Environment

• Reproduced on rc.5 (checkout) and rc.6 (npm) — same reader code family
• Both incidents: files re-compressed as a single zstd frame by an external repair script

[ZhYoo1226]

会话也是一个插件，文档中交代了会话日志的主旨是仅增加，不许修改。你们为什么会出现格式错误？手动改了第一帧吗

[bluewowo]

感谢回复！正文 "What we did wrong" 一节其实交代了原因，这里直接认账：
损坏不是 DSH 写路径产生的，也没有手改第一帧的内容——是我们自己的修复脚本在清理会话中的图块后，把整个文件重新压缩成了单帧 zstd（2896 行进 1 个 frame），违反了「首帧必须恰好一行 header」的契约。append-only 的意图是我们自己从带外操作破坏的，这点我们完全同意。
但想说明一下为什么会被迫去手工修：这个坏文件是 #2584 那个 replayState 毒化事故的产物（会话每轮回放必炸、fork 继承、无用户侧恢复手段），当时既没有 dsh doctor 之类的校验器，也没有隔离机制（quarantine），手工重写文件是唯一的恢复路径。帧布局的契约又完全没有文档（docs/persistence-catalog.md 没有），我们只能靠读 assertZstdHeaderFrame 源码反推出"每行一帧"的格式——踩了两次（一次打崩 web 实例、一次被插件吞掉）。
所以帖子里三个诉求的出发点其实是：append-only 设计本身没问题，缺的是"文件坏了之后"的恢复路径——

1. list() 隔离坏会话（跳过 + 大声记日志 + 可选 quarantine），别让一个文件打崩整个 host；
2. 把帧布局契约（header 帧恰一行 / 记录帧 / torn-tail 容忍）和"手工修复会话文件"步骤（重压为 header 帧 + 每行一帧、内容字节不变）写进文档，让下一个运维不用花一小时读源码；
3. （nice to have）dsh doctor 式的校验器，一条命令定位坏文件。
   这样手工编辑就从"唯一出路"降级为"有文档可依的最后手段"。谢谢！