v3→v4 session migration in 0.2.1-rc.2 fails to restore 0.1.5-rc.3 historical sessions (history invisible after upgrade)

[catsenior507]

Impact

After upgrading from 0.1.5-rc.3 to 0.2.1-rc.2 (SESSION_FORMAT_VERSION 3 → 4), opening any historical v3 session fails during the on-demand v3→v4 migration. The session list still renders (it reads only file headers), but opening a conversation shows no history — the migration throws and the session is treated as unreadable. The underlying files are intact on disk (94 v3 + 25 v0 + 7 v4, all header-readable); this is a migration failure, not data loss.

Root cause 1 — legacy plugin source encoding (systematic: 90/94 v3 sessions)

v3 sessions written by 0.1.5-rc.3 encode plugin message sources as:

{"kind":"plugin:@deepseek-ai/dsh-system-prompt"}

i.e. the plugin name is merged into kind with a plugin: prefix and there is no plugin field. The v3 codec in 0.2.1-rc.2 hard-requires the canonical form:

{"kind":"plugin","plugin":"@deepseek-ai/dsh-system-prompt"}

assertSystem in packages/.../dsh-session-format-v2-to-v3/lib/index.js (~line 263) throws SessionFormatError("system message requires plugin source") because source.kind !== "plugin". This is hit during v3 decode inside the v3→v4 migration, so every v3 session containing a system prompt fails to open.

Scan of all 94 v3 files on disk: 90 contain at least one system/message, and all 129 system sources use the broken {"kind":"plugin:@deepseek-ai/dsh-system-prompt"} shape (0 canonical). In total, 1827 message sources across 11 plugin names use the plugin:<name> form:

plugin name	occurrences
strict-gate	586
tool-jobs	494
@deepseek-ai/dsh-system-prompt	257
@dsh-external/dsh-client-plugin-context-assembler	212
tools-ptc	160
repeat-tool-reminder	40
dsh-session-title-llm	36
user-approval	30
tool-goal	4
compact	4
model-selection	4

The plugin:<name> string also matches the v4 producer-kind fallback in dsh-session-format-v3-to-v4 (producerKind returns plugin:${name}), which suggests 0.1.5-rc.3 was already emitting producer-kind-shaped sources into v3 files.

Root cause 2 — v3→v4 migration is stricter than the v3 reader on turn/step lifecycle (3/94 v3 sessions)

Three sessions are readable at the v3 level (verified with historicalSessionFormatCatalog.createRestore(...).finish(), which succeeds) but are rejected by the v3→v4 migration:

• 2 subagent sessions are missing the turn/end event for turn 1 (turn 2's turn/start follows turn 1's last step/end with no intervening turn/end). Migration error: turn/start does not open the expected turn.
• 1 session contains retroactive replacement tool/result events (surfaceOp ≠ append) referencing an already-closed turn, placed after a later turn/end. Migration error: tool/result is outside an open turn.

The lifecycle checks live in dsh-session-format-v3-to-v4/lib/index.js (the relationship validator, ~line 570 requireTurn and ~line 747 for turn/start).

Workaround I applied

A one-time script that normalizes "kind":"plugin:<name>" → {"kind":"plugin","plugin":"<name>"} in the v3 files (backing up the originals first) restores 91/94 sessions. The remaining 3 need the lifecycle-strictness issue addressed (or a targeted per-file repair I chose not to attempt, since it requires dense-seq renumbering with back-reference updates).

Suggested fix

1. In the v3 codec and/or the v3→v4 migration, accept and normalize the legacy {"kind":"plugin:<name>"} source form (treat it as {"kind":"plugin","plugin":"<name>"}, then map through producerKind as usual). This recovers the 90 sessions.
2. Make the v3→v4 migration's turn/step lifecycle validation at least as permissive as the v3 reader — either synthesize a missing turn/end when a turn/start opens a new turn, or accept retroactive replacement tool/results the way the v3 reader does.

Environment: Windows 11, @deepseek-ai/dsh-session 0.2.1-rc.2 (launcher dsh-web-launcher).

[catsenior507]

Additional root cause (distinct from the source-encoding bug above): single-frame v3 artifacts are silently skipped by list()

Investigating the same symptom on 0.2.1-rc.2, I found a second, independent reason history disappears — and it is a list-time skip, not an open-time migration failure.

Observation

• On disk: 111 session directories across 12 project dirs — 94 session.v3.jsonl.zstd, 9 session.v4.jsonl.zstd, 25 session.jsonl.zstd (v0).
• session/list returned only 24 sessions, with no error at all.

Root cause

The 87 missing sessions are all v3 artifacts written by 0.1.5-rc.3 as a single zstd frame (header + all events compressed together; measured frames == 1, frame 0 holds 117–1198 lines). The v4 reader's list path assumes the multi-frame layout.

readFirstZstdLine reads only the first frame and assertZstdHeaderFrame requires it to be exactly one header line:

function assertZstdHeaderFrame(plaintext) {
  if (plaintext.length === 0 || plaintext.indexOf(10) !== plaintext.length - 1)
    throw new Error("first frame is not exactly one header line");
}

A single-frame v3 artifact fails this check, readFirstZstdLine throws SessionPersistenceCorruptionError, and listArtifacts catches it and silently drops the session (metadata: undefined). That is why 87/111 sessions never appear in the browser sidebar.

Asymmetry that makes this a regression

The open/migration path (decodeStoredLog / prepareStoredMigration) uses the full scanZstdFrames multi-frame decode and reads single-frame v3 correctly. Only the list() header fast-path assumes the v4 multi-frame layout. So these v3 files are fully migratable — they are simply unreachable because the list never surfaces them.

Workaround applied (offline)

I rewrote the 87 single-frame v3 artifacts to v4 (header isolated in frame 0 + checksummed event frame) using the shipped format catalog:

createSessionFormatCatalogWithChildren([]).createRestore(header, { recovery: "recoverable", validation: "current" })
  → decodeRow(each) → finish() → encodeCurrentHeader / encodeCurrentEvent

After the rewrite, session/list returns 111 and every artifact passes first-frame validation.

Suggested fix

readFirstZstdLine (or the list header read) should not assume header-only frame 0 for pre-v4 generations — it should fall back to the full-frame scan that the migration path already uses.

[catsenior507]

从插件作者角度看 0.1.5 → 0.2.1：四处静默破坏，以及一份迁移指南的请求

我在这个版本上维护了四个 @dsh-external 插件（clock、context-assembler、command-center、web-watchdog）。升级后它们全部在某一处静默失败过 —— 没有编译错、没有启动错，只有运行时行为不对。下面每一条都是实测的，附最小复现线索。

1. kind: "plugin" 在 v4 里被直接拒绝，但没有公告

任何向会话注入消息的插件都受影响。v3 的写法 { kind: "plugin", plugin: "<pkg>" } 在 v4 中被 assertV4MessageSources 拒绝，整轮报 format v4 message requires a producer-owned source kind。正确写法是 { kind: "plugin:<pkg>" }（plugin 字段删除），也就是 producerKind() 的输出。

实测影响：一个广泛使用的第三方插件（dsh-policy-strict-gate 0.1.0）在每次写后检查时注入一条通知，于是每次改文件都会杀死当轮。用户侧看到的是"改个文件就崩"，而不是一条能定位的报错。

建议：这属于破坏性变更，值得在 release notes 里单列，并在类型上把旧形状标 @deprecated 至少一个版本。

2. v3 → v4 迁移路径对插件"隐形"

SESSION_FORMAT_VERSION 3→4 之后，单帧 zstd 的 v3 会话在 list() 里被静默跳过：readFirstZstdLine 只读首帧，assertZstdHeaderFrame 要求首帧恰好是一行 header，而 0.1.5 把整个日志压成了一个帧 → SessionPersistenceCorruptionError → listArtifacts catch 后不报错。本机 111 个会话只显示 24 个，且界面没有任何提示。

插件无法知道"这个会话为什么没出现"。建议 list() 返回被跳过的计数与原因，或至少打一条 warn。

3. dsh-jobs 的两处行为变更，只有读源码才知道

行为	后果
list() 不传 caller 只返回无主任务	插件看到 0 个任务，而任务明明在跑
read() 的消费游标按任务而非按调用者	第二个读取者（如状态监视器）会偷走 agent 的 job_output 输出

第二条尤其危险：它让"观察"变成"破坏"。0.2.x 新增的 readAt(id, from)（绝对偏移、不消费）是正确的解，但这一点没有任何地方说明，我是先踩坑、再读类型才发现的。

4. streamOpenTimeoutMs=3000

已在 #3413 报告，这里只作提醒：它是"标志性报错"的来源之一，同样不适合静默。

---

请求：一份 PLUGIN-MIGRATION.md

不需要很长的文档，但需要可操作。我建议的骨架（每项一行即可）：

变更内容 | 旧写法 | 新写法 | 是否静默失败 | 最小示例

具体到本次至少应包含：

1. 消息 source 形状（kind: "plugin" → kind: "plugin:<pkg>"）—— 附 producerKind() 的规则表；
2. 会话存储：v4 要求多帧 zstd，header 单独一帧；读取 API 与写入 API 的差异；
3. ctx.jobs：list/read/readAt 的可见性规则，以及"消费 vs 不消费"的语义；
4. ctx.schedule：它已经完整接管了提醒的创建与投递（这是好事！），但旧插件自建 store/engine 的迁移路径没有写；
5. ctx.skills.register、ctx.tools 等接口的类型变更。

一句总结

这次升级的设计方向是对的 —— ctx.schedule 接管投递后，我把整个自建引擎（约 45KB：store、scheduler、fire、PowerShell 任务镜像）删掉了，代码从 680 行降到 500 行，行为更可靠。

问题不在改动，而在改动的"发现成本"：四处破坏全部是运行时静默的，只能靠读源码或踩坑发现。一份迁移文档能把成本从"数小时"降到"数分钟"。

---

以上每条都有本机实测证据（会话日志、失败记录、复现脚本）；如果需要最小复现，我可以逐条补充。

[argszero]

Your second comment is the valuable half, and it holds up line for line at master 687f282e86 — but the asymmetry you describe is not quite what the source says, and your first root cause reads backwards. Both corrections change what the fix should be.

1. The list-time skip is real — and the open path fails too, with a different error

The silent drop is exactly as you traced it:

• readFirstZstdLine (packages/session/session-persistence-jsonl/src/index.ts:1497) decodes only the first structurally complete frame (:1513) and then requires it to be exactly one header line (:1516 → assertZstdHeaderFrame, :78: the only newline must be the last byte).
• Everything in that block is wrapped into SessionPersistenceCorruptionError (:1519-1523), whatever the cause was.
• listArtifacts catches precisely SessionFormatUnsupportedError | SessionPersistenceCorruptionError, substitutes metadata: undefined (:1148-1151), and continues (:1157). No warning, no count — your "87 of 111 vanish with no error" follows directly.

What does not hold is that the open path reads these files correctly. The decode path asserts the same invariant on the same first frame: readZstdPrefix (:993) scans all frames, decodes them, and calls assertZstdHeaderFrame(headerFrame.value) on the first decoded frame (:1016). For a single-frame artifact that first frame contains the header and every event, so the assertion fails there too.

The two paths differ in error typing, not in acceptance:

path	failure	observable result
list	wrapped in SessionPersistenceCorruptionError	silently dropped
open	the bare Error('… first frame is not exactly one header line') rethrown (:1071-1072)	loud, unexplained failure to open

So your offline rewrite was not a convenience around a list-only blind spot — with this reader those 87 artifacts are unreadable by both paths. Worth checking before you conclude otherwise: run one of the rewritten originals through the persistence backend's open path (not the format catalog) and see which of the two messages you get.

2. The fix is in the assertion, not in the list fast-path

Patching only readFirstZstdLine would surface the session in the list and then fail on open. The invariant that has to move is "frame 0 contains only the header" → "frame 0 begins with the header line":

• For a current (v4) artifact the strict form should stay, because it is the writer's own guarantee and it catches real corruption.
• For a pre-v4 generation the first frame is just a frame; the header line is its first line.

Both pieces already exist: readGenerationHeader has selected.sourceVersion in hand (:1179), so the tolerance can be gated on the generation the filename declares, and decompressZstdPrefix (session-persistence-jsonl/src/zstd.ts:154, already imported at :45) is the flush-what-is-there decoding that reads a prefix without demanding frame completion.

3. Root cause 1 reads backwards: the wrapper is v3 input, not the broken form

{ "kind": "plugin", "plugin": "<pkg>" } is not what the v3 codec wants instead of the correct form — it is the only form v3 can hold. At the tag you upgraded from, dsh-v0.1.5-rc.3, the session write admission itself enforced it: packages/core/session/src/index.ts:355-360 throws message must have plugin source unless kind === 'plugin' and plugin is a non-empty string. That generation had no way to append { "kind": "plugin:<pkg>" }.

The converter is built for exactly that wrapper. rewritePluginSource (packages/session/session-format-v3-to-v4/src/sources.ts:78) lifts {kind:'plugin', plugin} into the producer-owned shape via producerKind (:59): a rename table (:45) maps @deepseek-ai/dsh-system-prompt with role system to system-prompt, a second set (:51) keeps first-party identities under their own name, and anything else becomes `plugin:${plugin}` (:66). On the other side, v4 admission refuses the wrapper outright — kind must be a non-empty string and not 'plugin' (session-format-v3-to-v4/src/message-sources.ts:9).

Put together: stored v3 history needs no action from anyone; the migration does the lifting. What a third-party plugin must change is only what it writes at runtime on a v4 session. And the admission rule is narrower than "use plugin:<pkg>": any non-empty string other than the literal 'plugin' passes, so plugin:<pkg> is the convention the converter falls back to, not a requirement.

Which leaves a question about your 129 sources: if those files really declare v3 in their header and carry producer kinds with no plugin field, then 0.1.5-rc.3's session writer could not have produced them — its admission refuses that shape. Check one file's header version and mtime; whichever tool wrote them is the actual cause of that failure mode, and it is not the migration.

On the four-part migration guide

The shape you propose is the right one, and the mapping table above is the content of item 1 — but it has two audiences and they need opposite sentences. For stored history: nothing to do, converted automatically. For a plugin writing new events: supply a stable non-'plugin' kind, and expect the v4 admission error to name the failure if you do not. What is genuinely undocumented, and what your report makes worth having, is that second sentence.

[catsenior507]

Thanks — A and B are both right, and I re-verified A against the shipped bundle before replying.

A — confirmed. readZstdPrefix decodes frame 0 and calls assertZstdHeaderFrame on it, so a single-frame v3 artifact fails the open path too; the two paths differ only in error typing (silent drop vs. loud unexplained error). My earlier "asymmetry" framing was wrong.

B — agreed. The invariant to relax is in the assertion ("frame 0 begins with the header line" for pre-v4), gated on selected.sourceVersion, not in the list fast-path. Patching only readFirstZstdLine would surface the session and then fail on open.

C — the on-disk artifacts say otherwise. I ran the shipped catalog migration against an original (pre-my-edit) v3 file before replying. It fails:

• header: {"type":"session","version":3,"id":"…","createdAt":1786865124971,…} — createdAt ≈ 2026-08-16, squarely the 0.1.5-rc.3 lifetime
• its system/message events carry data.message.source = {"kind":"plugin:<pkg>"} — producer form, no plugin field
• result: SessionFormatError: system message requires plugin source

The throw is assertSystem at dsh-session-format-v2-to-v3/lib/index.js:263, called from assertV3StructuralRow (:239) — i.e. the v3 decode admission:

if (source["kind"] !== "plugin" || typeof source["plugin"] !== "string" || source["plugin"].length === 0)
  throw new SessionFormatError("system message requires plugin source");

So the v3 decoder rejects producer form before the v3→v4 rewritePluginSource you cite can lift anything — the converter never runs for these rows, because the decode admission already threw.

This is not a stray file. Across all 90 original v3 artifacts the plugin sources are uniformly producer form — 1,827 occurrences, 11 distinct plugins, zero wrapper-form:

kind	count
plugin:strict-gate	586
plugin:tool-jobs	494
plugin:@deepseek-ai/dsh-system-prompt	257
plugin:@dsh-external/dsh-client-plugin-context-assembler	212
plugin:tools-ptc	160
plugin:repeat-tool-reminder	40
plugin:dsh-session-title-llm	36
plugin:user-approval	30
plugin:tool-goal	4
plugin:compact	4
plugin:model-selection	4

None of them carries a plugin field. Normalizing producer → wrapper is what made the migration pass (those 1,827 replacements).

On your provenance question — it's a fair probe, and I did check. Header version is 3 and createdAt is mid-August 2026, so "another tool after the upgrade" is ruled out. What it actually points to is that every one of those 11 sources is a plugin name, i.e. plugin-injected system/message rows: the most consistent reading is that under 0.1.5-rc.3 the plugin message-injection path wrote the producer form and did not go through the packages/core/session write admission you cite, while the 0.2.1-rc.2 v3 decoder (assertSystem) demands the wrapper. That is a write/decode asymmetry in the v3 source shape — not a third tool, and not "the migration does the lifting for free."

Either way, "stored v3 history needs no action" doesn't survive contact with the actual artifacts: as-is they throw during migration, and it is specifically the wrapper normalization that unblocks them.