[Docs proposal] document the multi-frame zstd session-log container format

[ciceroyang]

Proposal: document the session-log container format (concatenated zstd frames) in the official docs.

Why this matters
The on-disk session artifact is a concatenated-frame zstd container, not a single zstd stream. This is load-bearing for the ecosystem but currently undocumented: the one-shot zstdDecompressSync (and the zstd CLI's naive single-frame paths) silently yield only the FIRST frame — the header line — and drop every event. Every community tool that reads historical sessions hits this wall independently.

Evidence it is already biting

• dsh-doctor's log_health check exists precisely because of this (frame scan + full decode; discussion 💡 Idea: Add 'dsh doctor' CLI command for environment & dependency diagnostics #1719)
• dsh-report-studio's cross-session weekly (report_week) had to port scanZstdFrames from dsh-session-persistence-jsonl to read history
• dsh-trajectory (session → HTML trajectory) does the same
• the community tutorial documenting the pitfall: https://github.com/ciceroyang/dsh-report-studio/blob/main/docs/tutorial-zh.md (坑 6)

The ask
Add a short section to the session-persistence docs (or the architecture doc's Session log section) stating:

1. the artifact is a concatenation of independently decodable zstd frames
2. consumers must either scan frame boundaries structurally (the scanZstdFrames algorithm in dsh-session-persistence-jsonl/format.ts is the reference) or stream-decompress
3. one-shot decompressors stop at the first frame by design

One paragraph of docs saves every future session-tool author the same multi-hour debugging session. The reference algorithm already exists in the repo; this just makes it discoverable.

[zoahdev]

We hit this exact wall building dsh-replay — a one-shot zstdDecompressSync returns only the first frame and silently drops every event, which is precisely the multi-hour debugging trap this proposal describes.

A working reference implementation already exists and is tested:

• engine/replay.js — scanZstdFrames walks the container structurally: validates the FD2FB528 magic, parses the frame header (single-segment + frame-content-size), slices each frame, and feeds each one to zstdDecompressSync. It also tolerates a torn final frame (a crash mid-write) instead of throwing.
• The README notes the decoder mirrors upstream scanZstdFrames + decodeStorageRecord.

I also keep a docs-proposal home at dsh-docs. Happy to add a short "session-log container format" section there (concatenated zstd frames → scanZstdFrames reference → one-shot-decompress pitfall) and mirror it as an official docs PR once the channel opens.

tl;dr — this is a real ecosystem trap, and the reference algorithm this proposal asks to document is already shipped + tested in dsh-replay.

[ciceroyang]

Excellent corroboration — with dsh-replay that makes three community tools that independently reverse-engineered the same container (dsh-doctor's log_health, dsh-report-studio's report_week, and your replay engine). The proposal's evidence section just got stronger: this is not a niche trap, it is the standard onboarding wall for session-tool authors.

Two things to fold into the proposal from your comment:

1. Torn final frame: our scanZstdFrames ports stop at the incomplete frame; your replay engine goes further and tolerates it (decompresses the prefix instead of throwing). The docs section should document both tiers — structural scan (complete frames) and crash-tail recovery (torn frame) — because a reader building an audit tool needs the first, and a reader building a repair tool needs the second.
2. dsh-docs as the staging home: +1. Publishing the section there now (with the official-docs mirror noted as pending) means the knowledge stops being tribal immediately, which is the point of this proposal.

Suggested canonical section content: artifact = concatenated zstd frames; consumers scan boundaries via scanZstdFrames (upstream format.ts reference) or stream-decompress; one-shot decompressors stop at the first frame; torn final frames occur after crashes and may be prefix-decoded.

Happy to review the dsh-docs section once you draft it, and to keep #2328 open as the upstream ask until the official docs pick it up.

[zoahdev]

Drafted it — docs/specs/session-log-format.md in dsh-docs, with the two tiers you called out: structural scan (complete frames) and crash-tail recovery (torn final frame → prefix/skip instead of throw).

The "one-shot decompressors stop at the first frame" trap is stated as rule #2, and the page names the three tools that hit it (log_health / report_week / dsh-replay) so the next author reads instead of reverse-engineers.

Pull requests / comments welcome — I'll keep #2328 as the upstream ask and mirror into the official session-persistence docs when the channel opens.

[ciceroyang]

Reviewed the draft — accurate and exactly the right shape. Endorsed, with three precision additions worth folding in so the next author still doesn't need the upstream source:

1. Tier 1 must skip the optional fields, or frame ends misalign. After the frame-header descriptor:

   • window descriptor (1 byte, absent when single-segment)
   • dictionary id (0–4 bytes per descriptor bits 0–1)
   • frame content size (per bits 6–7; for FCS flag 0 it exists only when single-segment)
   • and if the checksum flag (descriptor bit 0x04) is set, a trailing 4-byte content checksum after the last block.
     The upstream scanZstdFrames handles all of these; a hand-rolled scanner that skips only the descriptor will desync on the first checksummed frame.

2. Block payloads of type RLE (0x01) are one byte, not blockSize. The block header's size field is literal for raw/compressed blocks but the RLE type stores a single repeated byte — the upstream loop has the payloadBytes = blockType === 1 ? 1 : blockSize branch. Easy to get wrong the same way twice.

3. A one-line pointer to the interior format would close the loop. Inside the frames it is JSONL: a {"type":"session",...} header line, then event lines with {type, seq, time, data}, plus packed chunk rows that use seq0 instead of seq. One sentence ("interior JSONL layout: see the event-log docs") keeps container and content docs linked.

With those in, the page fully replaces the upstream source for a new implementer. Thanks for drafting it — this turns #2328 from a proposal into an artifact.