Small wins for the session layer: zstd level by default (+ measured numbers), and corpus-gated acceptance for the #4416 roadmap

[MedicineKing]

Small wins for the session layer: zstd level by default (+ measured numbers), and corpus-gated acceptance for the #4416 roadmap

Two small, previously-unmentioned wins in one proposal. Both are deliberately split from the big threads (#4416's roadmap, #4942/#4945's write-side work) because neither belongs there and both need no code archaeology to land.

Part 1 — zstd: the JSONL backend compresses with defaults, and the numbers say a lot

The asymmetry no one has mentioned. session-persistence-sqlite ships a tuned data dictionary (resources/zstd-dictionary.bin, loaded in compression.ts:42-46) and feeds it to every compress/decompress. session-persistence-jsonl compresses with CHECKSUM_OPTIONS = { ZSTD_c_checksumFlag } and nothing else (zstd.ts:18-19) — no level, no dictionary. Session logs are textbook zstd material (JSON structure + recurring system-event templates), so I measured the gap on real artifacts:

Config	26.4 MB text corpus	4.3 MB text corpus	bytes saved vs default
default (current)	5.56 MB / 67 ms	0.795 MB / 11 ms	—
level 9	5.01 MB / 328 ms	0.708 MB / 45 ms	~11%
level 15	4.81 MB / 3.3 s	0.688 MB / 323 ms	~15%
level 19	4.04 MB / 7.6 s	0.588 MB / 975 ms	~37% (×100 latency)
reuse SQLite dict + level 9	5.23 MB / 363 ms	0.723 MB / 55 ms	~6% saved, but still behind level 9's 11% — the SQLite-tuned dictionary does not transfer to session JSONL

Conclusions:

• compressionLevel: 9 is the sweet spot: ~11% smaller storage on whole-log measurement. Because the backend compresses per append batch (not per whole log), I measured the batch path directly: 100 turn-sized batches (median 60.9 KB/batch), 5 runs each, median-of-medians — default 0.25 ms/batch, level 9 1.14 ms/batch, i.e. +0.9 ms per batch for ~6.6% smaller batches, which is same-order as the whole-log result. Write-latency impact at the per-batch level is therefore sub-millisecond to low-single-digit ms, not the whole-file 328 ms in the table (that is cold-rewrite worst case only). And for context: fsync dominates the per-batch write path (appendLines = write + handle.sync() per batch) — a +0.9 ms compression delta is within its jitter. Read path is untouched (level affects the writer only).
• Old logs stay readable: zstd frames do not record the compression level, so changing the default affects new writes only — existing artifacts decode identically.
• Reusing the SQLite dictionary does not transfer (it is tuned for SQLite rows; measured ~6% saved — still behind level 9's ~11% — and is not worth the decoder sidecar it requires). A JSONL-specific dictionary would require train + dual-side decoder changes plus frame-compat work — low ROI for an uncertain ~4% over level 9; I would not pursue it.
• level 15-19 trade 3-100x latency for ≤1.5x the gain of level 9 — not worth a default; a config knob covers the rare archive-size-sensitive deployment.

Suggestion: make the JSONL compression level configurable with 9 as the new default (parameterize CHECKSUM_OPTIONS; one-line change plus a setting). Store-side win with writer-side and reader side both safe. (Measurement rig: zstdCompressSync over real-session text, warm run, medians of 5; per-batch rigs use 100 turn-sized batches — happy to share the script.)

Part 2 — verify-first: every stage of the #4416 roadmap should land behind the #4945 corpus gate

The #4416 roadmap (7 stages A-G: tail-index sidecar + JSONL seek, frame-merge, detached projection, snapshot-on-demand, header cache, streaming pagination, default SQLite / format evolution) is an excellent, measured plan — but none of its stages defines its acceptance criteria against the corruption corpus. Every stage rewrites or bypasses parts of the read path — which is precisely the most fragile surface in this codebase (the corruption family across #1333…#4662 lives there).

Proposal: bind each stage to the #4945 regression gate before merge:

• every A-G stage ships with a corpus run: all known shapes replayed through the official read path, expected outcomes unchanged (partial-tail → ACCEPT with the tail trimmed to the last committed boundary — the crash remnant is dropped by design; two-writer → REJECT; etc. — the expectation table is already published);
• plus the stage's own new boundary case added to the catalog (e.g. the tail-index stage adds an "index vs artifact drift" row);
• the per-artifact table from the gate becomes the merge-review attachment, exactly as proposed in Regression gate for the session layer — a corruption corpus that would have caught this entire bug family #4945.

I can run the first pass of the gate on any candidate stage (the pipeline is single-command, zero install, official-path referee: Session.fromRestore / assertZstdHeaderFrame / scanner contract — all unmodified npm 0.1.1-rc.2) and publish the before/after tables in the thread.

What this costs

Part 1: one-line parameterization + a default change (plus the measurement methodology above if you want reproducibility). Part 2: a review habit plus the existing gate. Both are gated on the same referee, so neither can silently break the committed region.

[denial123789]

The handbook now carries a reusable framing and compression-level evaluation guide: https://github.com/sandbaseai/deepseek-harness-handbook/blob/main/docs/en/architecture/session-framing-performance.md. It separates whole-log measurements from per-batch latency, treats level 9 as a measured tradeoff rather than a universal default, and keeps every writer change behind the #4945 corruption-corpus gate. Independent community documentation; upstream remains the implementation authority.