[Docs] Generated persistence catalog omits the packed chunk rows that packChunks writes to the log

[doctutor]

docs/persistence-catalog.md describes itself as covering

Every event type that can appear in a session's durable event log

but with packChunks enabled — the default — the JSONL backend also writes packed storage rows that the catalog does not list. At 99f6f02fe (dsh-v0.1.0-rc.7) I see three: text-chunks, reasoning-chunks, and tool-call-chunks. Searching the generated catalog for those three names returns nothing.

They are also shaped differently from a SessionEvent: they carry seq0/time0 rather than seq/time, and pack a run of deltas into arrays. An example line, lightly trimmed:

{"type":"text-chunks","seq0":10,"time0":1787150994000,
 "data":{"turn":1,"step":1,"index":0,"dt":[21,1,0,0,20],
         "texts":["He","llo",", ","wor","ld"]}}

Repro

1. Run a session with persistenceCompression: none (so the log is readable as text) and packChunks left at its default.
2. Ask for a reply long enough to stream several deltas.
3. grep '"type":"text-chunks"' <persistenceRoot>/**/session.jsonl — present, and absent from docs/persistence-catalog.md.

Why it matters

The catalog is generated from source, which is exactly what makes the gap easy to trust into: a reader implementing a log parser against it has good reason to believe it is complete, and then meets rows it cannot classify. The failure is quiet in a specific way — a parser that walks chunk events to reconstruct assistant text will silently miss every packed run, and on a thinking-enabled session that can be most of the output, with no error anywhere.

I hit this from the safe side (keying on seq, which packed rows lack), but only by accident of how the reader was written.

Related

This is the same family as #2328 ([Docs proposal] document the multi-frame zstd session-log container format) but one layer in: that one is about how the file is framed, this one is about which row types appear once you are past the framing. A single "what is actually in a session log" docs section could close both.

Suggestion

Either extend the generator to cover the storage-row encodings, or narrow the catalog's opening sentence to say it documents SessionEvent types specifically and point to the persistence docs for what else the log file can contain. The second is cheaper and would have been enough to prevent the wrong assumption.

[argszero]

Verified at rc.8 (141eb6fef8) — the gap is exactly as described, and it's worth closing properly rather than just narrowing the catalog's opening sentence.

Source confirmation. session-persistence-jsonl/src/format.ts:221-223: eventLines runs packChunks ? packChunkRuns(events) : events — so with the default (DEFAULT_PACK_CHUNKS true, index.ts:128/157) the JSONL backend writes StorageRecords, not SessionEvents. The three names (text-chunks/reasoning-chunks/tool-call-chunks) come from packChunkRuns in the shared @deepseek-ai/dsh-session package (StorageRecord + decodeStorageRecord live there too, and scanLog decodes rows layout-blind — format.ts:351). The generator (scripts/gen-persistence-catalog.ts:2) walks SessionEventMap declarations only, so the storage-row vocabulary is structurally invisible to it. Your example row shape (seq0/time0/dt/texts) matches packChunkRuns' delta-run encoding.

Why the "narrow the sentence" option is cheaper but weaker. The failure you hit (a parser keying on seq silently missing packed runs) is a reader-classification failure: the reader meets rows it cannot classify, and there is no error. Narrowing the catalog's claim to "SessionEvent types" makes the docs honest, but the reader still has to discover the packed vocabulary somewhere else, and on a thinking-enabled session packed runs are most of the output. Since StorageRecord is a small closed set (one union in the shared package), the generator can enumerate it — option 1 (extend the generator) is only modestly more work and removes the whole class of silent-miss.

A third option that closes both the catalog gap and the family root. Rather than only listing the storage rows as a static vocabulary, the generator could emit the mapping event shapes → storage shapes: which SessionEvent kinds packChunkRuns absorbs (the delta-chunk events), how they map onto seq0/time0/dt arrays, and the invariant that decodeStorageRecord restores the original events losslessly (the "reading is layout-blind" contract, format.ts:218 comment). That turns the catalog from a type list into a round-trip contract — which is also exactly what a log-parser implementor needs.

Family note. This connects to the session-log corruption family (#3198 and friends): the seq0/time0 storage-row shape is a second vocabulary where the seq-continuity assumptions in strict scanners (format.ts committed-region check) can silently misalign. A catalog entry that documents "packed rows carry seq0 (first seq of the run), not seq" is the cheapest guard against the next parser that, like the reporter, keys on seq and treats packed output as absent.

Also agree with the #2328 linkage — one "what is actually in a session log" section covering framing + row types would be the right home for both.

[denial123789]

I turned the rc.8 source findings in this thread into a reader-oriented format map: https://sandbaseai.github.io/deepseek-harness-handbook/deepseek-harness-session-log-format.html

The key contract is now explicit there:

• .jsonl.zstd is a concatenation of one checksummed header frame plus append frames, not one monolithic frame;
• the post-header vocabulary is SessionEvent | ChunkRow, not only the generated SessionEventMap catalog;
• text-chunks, reasoning-chunks, and tool-call-chunks carry seq0/time0 and expand to multiple events;
• every row should pass through the shipped decodeStorageRecord codec before contiguous-seq validation; and
• packChunks: false changes only future writes, while reading packed rows remains unconditional.

The guide includes a safe read-only inspection sequence and eleven parser acceptance gates, including mixed-layout equivalence, malformed-row failure, committed-region gap detection, and a packChunkRuns → JSON → decodeStorageRecord deep-equality round trip. All version-sensitive links are pinned to rc.8 commit 141eb6fef8.

This is a community-maintained operational reference, not an upstream format guarantee; unsupported format versions should still fail explicitly.