Zstd session log: undocumented one-header-line first frame, misleading error, unusable recoverable mode

[AnnidLin]

3/3 - Zstd session log: undocumented "first frame must be exactly one header line" rule, misleading error text, and a recoverable read mode that still throws

Version: 0.2.1-rc.5 (Windows desktop)
Area: @deepseek-ai/dsh-session-persistence-jsonl

1. The frame rule is strict, undocumented, and silently enforced

assertZstdHeaderFrame (lib/index.js:2298):

if (plaintext.length === 0 || plaintext.indexOf(10) !== plaintext.length - 1)
  throw new Error("corrupt Zstandard session log: first frame is not exactly one header line");

readFirstZstdLine (:3379) reads only the first frame and applies that assertion; listArtifacts (:3103) catches the resulting error and skips the artifact.

Consequence: a session log whose header line and events are re-compressed into a single zstd frame is byte-for-byte correct as plaintext, passes validateStoredEvents on the full event list, and is still rejected - because the physical framing is wrong. Nothing in the documented format says a frame boundary is part of the contract.

We hit exactly this while hand-repairing a corrupted log: the repaired file validated PASS, events 711 and then vanished from the list after restart.

2. The error message points at the wrong thing

The message says "corrupt Zstandard session log" for a file whose zstd stream decodes perfectly. It should say what actually failed: the first frame contains more than the header line. As written it sends the reader hunting for a compression bug that does not exist.

3. recoverable mode is not actually recoverable

scanRows accepts recovery === "recoverable", suggesting a degraded load is available, but it still throws when it encounters turn/end (and the other relationship checks). So there is no code path that loads a log with a single bad event - the only options are "loads fine" or "gone".

Suggested fix

• Either accept a multi-line first frame (decompress the first frame, take line 1 as header, treat the rest as events), or emit an explicit, distinct error such as first frame must contain only the header line.
• Make recoverable actually degrade: skip/quarantine offending events and load the rest, reporting what was dropped.
• Document the frame contract, since anyone writing a repair tool necessarily produces this file format.

Repro

1. Take a valid session.v4.jsonl.zstd, zstd -d it to h.jsonl (712 lines), recompress the whole file as one frame.
2. validateStoredEvents(meta, events, loc) on the decoded events: PASS.
3. Restart, open the workspace: session absent; forcing it open yields the first frame is not exactly one header line corruption error.

---

Appendix - the three physical variants of the same 712 lines

variant	frames	bytes	decodes to	loads?
original (corrupted by the bad append)	455	283,627	712 lines / 683,441 B	no - relationship error
hand-repaired, recompressed as one frame	1	123,780	712 lines / 731,343 B	no - "first frame is not exactly one header line"
DSH's own layout (header frame + body frame)	2	123,930	712 lines / 731,343 B	yes

The middle row matters: validateStoredEvents(meta, events, location) returns PASS, events 711 for it - every event is well-formed - and it is still rejected by the loader because of a frame boundary that no document mentions.

---

Related: #12007 (how the log got bad) and #12008 (why the session then vanished).

The thing a reader needs to judge this report: validateStoredEvents returning PASS does not imply the artifact loads. We measured the 1-frame variant of a 712-line log: PASS, events 711, and the loader still rejected it with first frame is not exactly one header line.

So there are two independent acceptance criteria - event-level validity and frame layout - and only the first one is discoverable from the public API. Anyone writing a repair tool (or any future code path that re-compresses a log) will hit the second one blind.

[PerryLink]

The framing rule is documented, not just enforced.

packages/session/session-persistence-jsonl/README.md:111 says it in as many words:

The default artifact is a standard concatenation of independent Zstandard frames: one checksummed frame containing only the header line, then one checksummed frame per durable append batch.

So a log whose header line and events live in one frame is not a valid artifact with an undocumented constraint — it is a different physical layout. The check you found (src/index.ts:78-80, called from :1018 and from readFirstZstdLine at :1499/:1519) is enforcing the documented layout, and list() omitting such headers is also documented (README.md:91: "Unreadable or unsupported headers, including corrupt Zstandard header frames, are omitted from discovery and list()").

That said, two of your points survive the correction:

1. The error text is wrong for this case. A layout violation is reported as corrupt Zstandard session log, which sends people looking for checksum damage. It should say the header frame must contain exactly one line, and name the file. Same string is duplicated in generation.ts:630, so it is one wording change in two places.
2. Relaxing it is a format decision, not a bug fix. Accepting header+events in one frame would make two physical layouts valid, which the same README paragraph rules out; that needs a migration story, not a patch.

I could not verify your third point (the recoverable read mode still throwing) without your exact call — if you post the invocation and the error, I can trace it at fcc0ad46.

For recovery tooling in the meantime: repack with the header line in its own frame, then the event frames — that is the shape list() expects.

Read-only source inspection at master fcc0ad46; I did not run your repacked log.