[Bug] Cold history loading fully materializes large/corrupt logs and can stall the entire Web server

[wellorbetter]

Summary

Opening a cold, very large session calls the persistence inspection path before response pagination is applied. The server therefore decompresses, parses, validates, and materializes the complete event stream on the Node.js main thread even when the client requests only a bounded history page.

In a field incident, one structurally valid large session and one session with a committed seq rollback made dsh web accept TCP connections on 127.0.0.1:3080 while even GET / timed out. Quarantining only those session trees and restarting restored normal response times.

The writer-corruption mechanism is already tracked in #1433, #1452, and #1497. This report is specifically about the server-wide availability and repeated full-scan amplification when the client opens one large or corrupt cold history.

Environment

• Windows 11
• Node.js v24.14.1
• @deepseek-ai/dsh@0.1.0-rc.6
• dsh web
• Source inspected at 47f943859bef60e4160492346772ded9b24f765a

User-visible behavior

Startup printed the normal address:

dsh web: http://127.0.0.1:3080

The process was listening, but an HTTP request did not complete:

HTTP_ERROR after 10185ms: The operation has timed out.

During the stall:

node working set : approximately 163 MB
CPU delta        : approximately 0.95 CPU seconds / 2 seconds wall time
socket state     : multiple CLOSE_WAIT connections accumulated on port 3080

The corrupt history RPC returned:

history unavailable for session "…":
Error: corrupt session log: seq gap in committed region at line 12804
(expected 187372, got 187368)

Default persistent logs did not capture this failure. The Web stdout log contained only the bound URL and stderr was empty. The error was available only in the browser/RPC response and was later confirmed deterministically from the raw session artifact.

Session evidence

The two private session IDs and raw artifacts are intentionally omitted because they contain user conversation data.

Session A: corrupt committed history

compressed size : 4.853 MB
decoded size    : 16.587 MB
zstd frames     : 9,730
storage records : 26,934
expanded events : 461,981

The durable event boundary was:

249437 tool/call
249438 tool/result       synthetic interrupted-tool recovery
249439 step/end          synthetic recovery
249440 turn/end          synthetic recovery
249441 session/end-seed
249438 tool/result       stale continuation; seq moves backward by four
249439 step/end
249440 step/start
...

Independent scanning reports expected 249442, got 249438. An older backup of the same session contained the same N -> N-4 shape (expected 187372, got 187368). This matches the already reported overlapping/stale-writer family in #1433, #1452, and #1497.

Session B: structurally valid but very large

compressed size          : 6.984 MB
decoded size             : 15.971 MB
zstd frames              : 21,153
storage records          : 27,030
expanded events          : 444,316
turns / steps            : 50 / 482
child sessions           : 12
projected surface context: approximately 601,559 tokens

This session has no sequence gap. It reproduces the excessive cold-history/resume work independently of structural corruption.

Source-level cause

For a cold session, historySourceFor() calls sessionPersistence.inspect(sessionId) and receives the full event array:

• api-proxy.ts lines 1533–1538

Only after that complete inspection does historyPage() apply paginate(events, request.maxMessages, ...):

• api-proxy.ts lines 805–818
• api-proxy.ts lines 2242–2259

The client requests history immediately when opening a conversation:

• session.ts lines 614–642

The error handler converts an inspection failure into an RPC internal response but does not persist a structured server-side diagnostic:

• api-proxy.ts lines 2260–2269

As a result, maxMessages bounds the returned payload but does not bound persistence I/O, Zstd decompression, JSON parsing, validation, or event materialization. Reopening or retrying the selected conversation repeats the expensive work. Because this work shares the Web server's Node.js process, unrelated requests can become unresponsive.

Reproduction

A safe synthetic repro should use an isolated temporary DSH_HOME:

1. Create/import a valid multi-frame Zstd session with hundreds of thousands of expanded events.
2. Start dsh web and open that cold conversation.
3. Concurrently measure GET / latency and event-loop responsiveness.
4. Repeat with a committed backward-sequence block near the end of the artifact.
5. Refresh/reopen the conversation several times.

Actual result: history loading repeatedly performs full inspection; the selected history may fail, and unrelated HTTP requests can stall.

Expected result: bounded history reads should do bounded work; one corrupt session should fail in isolation without affecting the rest of the Web server.

Suggested fixes

1. Add a persistence API for bounded tail/history reads by message boundary or sequence range.
2. Maintain an index/checkpoint from message or sequence boundaries to Zstd frame offsets, and apply maxMessages while reading rather than after full materialization.
3. Negative-cache a corruption result by immutable artifact revision so repeated UI retries do not rescan the same file.
4. Move large decode/replay work to a worker thread or process it in cooperative bounded slices.
5. Persist one structured corruption diagnostic per artifact revision under $DSH_HOME/logs, including session ID, record/frame position, expected/actual seq, and stack trace.
6. Provide non-destructive dsh session verify / dsh session quarantine commands.

The correctness fixes proposed in #1433/#1452/#1497 (cross-process writer ownership and durable-tail validation) remain necessary; the measures above limit the blast radius when corruption or unusually large histories still occur.

Workaround verification

The two affected root sessions and their descendants were moved out of the active session root, not deleted. After one clean restart:

GET / first request : HTTP 200 in 243 ms
GET / warm request  : HTTP 200 in approximately 17 ms
CPU over 2 seconds  : 0.02 CPU seconds

Other histories remained available. This A/B result isolates the outage to active history discovery/loading rather than installation, networking, or the static Web bundle.

[argszero]

Confirmed against master — and the "server-wide availability" framing is the important part. This is the availability amplifier of the corruption family I've been tracking all day (#1333/#1452 seq gaps, #1473 bad frame, #1497 replayed tail): corruption alone makes one session unreadable, but the full-scan-on-cold-load behavior turns it into a whole-server stall.

Source confirmation

packages/session/session-persistence-jsonl/src/index.ts:

• readPrefix (:310) → for zstd calls readZstdPrefix (:348) which decodes every frame in the artifact; for plain JSONL calls scanLog which parses the complete buffer. Both materialize the entire event stream.
• readStableFile (the read the inspector uses) re-reads/verifies the whole file (:283-299 in the revision loop).
• The corruption tail you show (synthetic interrupted-tool recovery + session/end-seed + stale continuation with seq moving backwards) is exactly the Bug: corrupt session log after unclean exit — replayed events cause seq gap in committed region, history unrecoverable #1497 "replayed committed events" mechanism — same signature, one more byte-level sample for that family.

So a cold load() of Session A (461,981 events, 16.5 MB decoded) does a full decode+parse+validate on the Node main thread before any pagination applies. Your CLOSE_WAIT accumulation + CPU pinning is the predictable result. The listArtifacts path is cheap (first frame only), but the history RPC is not.

Why this is worse than the corruption alone

The four-thread family has a consistent theme: one bad log = disproportionate blast radius. Here the blast radius isn't just "session unavailable" — it's GET / times out for everyone because the main thread is saturated. That's a DoS-by-corrupted-session, and it needs a different fix layer than "repair the log":

1. Bound the read: cold history should not require full materialization. A page of history only needs a bounded window of events — the persistence layer needs a seekable/bounded read (read frames incrementally, stop at the page boundary) instead of decode-everything-then-slice.
2. Detect corruption at the cheap scan, not after full parse: the seq-gap is visible from a frame-level scan long before 461k events are materialized. The loader already does a frame scan for the header; extending that to a "seq monotonicity scan over frames" would let a cold load fail fast and cheaply instead of decompressing everything first.
3. Quarantine at list/inspect time: per the community's session-doctor proposals (Add a session doctor and versioned migrations #1518, dsh-doctor: an offline diagnostic for the failure classes this community keeps reporting #1534), an unreadable session should be quarantined/reported at listArtifacts time so it never reaches the full-scan path. Add a session doctor and versioned migrations #1518's acceptance criterion ("an unreadable session does not prevent the rest of the workspace from loading") should extend to "does not stall the rest of the workspace".

Suggested regression shape

A test that opens a cold session with a large/corrupt artifact and asserts the inspect/history call either (a) returns within a bounded number of decoded frames, or (b) fails fast with a cheap scan error — never full-materializing. Combined with the existing corrupted-log fixtures, this would have caught today's incident.

This also feeds the S6/S7 checks in moonquake2004's dsh-doctor (#1534) — a frame-level seq scan is exactly the offline check I proposed there, and it doubles as the runtime fail-fast path. Happy to help sketch the bounded-read design against readPrefix if useful.

[moonquake2004]

The offline side of this is now covered — for anyone who wants to identify which sessions to quarantine without starting dsh:

• S6/S7 (in dsh-doctor.mjs, shared in dsh-doctor: an offline diagnostic for the failure classes this community keeps reporting #1534) detect the Session A signature: I tested the exact N → N-4 shape (end-seed, then a stale continuation with seqs moving backwards) — S6 flags the seq gap and S7 flags the post-end-seed replay, both firing on a synthetic replica of your corruption tail. So the "one bad log" can be found purely offline, before it ever stalls the server.
• New S11 — whole-session scan: enumerates every session under $DSH_HOME/sessions, and for each runs the cheap frame/seq checks + the size materialization metrics, then reports two buckets:
  • corrupt (S6/S7/S8/zstd failures) → quarantine suggestion: move the session dir out of $DSH_HOME/sessions;
  • oversized (decoded >10MB or >10k frames — your Session B class) → cold-open materialization-risk warning (informational, not a hard failure).
    This is the "quarantine at list/inspect time" half of argszero's fix list, done as an offline stopgap until the persistence layer gets a bounded read.
• Note: on a machine with a very large live session the oversized warning fires legitimately (cold-opening it is genuinely heavy) — the check is honest about that rather than hiding it.

Thanks for the field numbers — they calibrated the thresholds.

[7889545]

Supplementing with a reproduction that needs no corruption at all — pure scale, and the stall is boot-wide, not only on cold-open.

Boot materializes ALL sessions, valid or not

With 15 valid sessions (several 1.8–2.7 MB compressed / ~300k tokens / 5–7k events each) in sessions/<cwd>/, dsh web climbs to ~3 GB heap within 60–90 s of boot with zero client interaction — no browser, no history RPC. Then GET / stalls (0.2 s sometimes, 6–20 s timeouts, alternating).

• CPU profile (20 s, --inspect + CDP Profiler): ~60% of samples in structuredClone and its internals; hot JS frames Session.create/fromRestore (:1355/:1367), snapshotSessionEvent (:1172), adoptSessionEvent (:1153) in dsh-session; readSession (:811), consumeEventLine (:275) in dsh-session-persistence-jsonl.
• Heap snapshot (heapUsed 1.72 GB): 21.6M live objects; top class-named: Object 526 MB, string 329 MB, assistant/chunk 59 MB, reasoning-delta 38 MB, tool-call-delta 17 MB, text-delta 4 MB — i.e. the session event set, all materialized.

Workspace archiving does NOT reduce this

Tested on the same session set:

State	Boot heap ~90 s	GET /
15 sessions, default config	~3 GB, climbing	times out
preparedSessionCacheSize: 1	~2.7 GB	times out
11/15 moved to archivedSessionIds (workspace archive)	~3 GB	times out
11 session dirs moved out of sessions/<cwd>/	~306 MB, stable	0.2 s, consistent

So the materialization surface is the sessions directory, adopted in full at boot, regardless of the workspace archive set and of preparedSessionCacheSize. This is broader than the history-RPC path argszero scoped: the same bounded-read / lazy-adoption fix should apply to boot-time adoption, not just cold-open. A cheap corollary: archivedSessionIds currently hides from display but still costs full memory — worth making it a real materialization boundary.

Verification method: WorkingSet64 sampling of the web PID across boot; CPU profile + heap snapshot captured via --inspect/CDP; memory correlated with the session set.

[argszero]

Verified the cost cascade and one config surprise against master (47f9438) — your field numbers are the per-session amplifier in action, and there are three details the thread doesn't have yet:

1. Why preparedSessionCacheSize: 1 didn't move the needle — it's a retention knob, not a materialization gate.
The config JSDoc is explicit: "Maximum completed unpublished preparations retained for reuse" (session-persistence/src/coordinator.ts:79-80, default 5, min 1). It caps how many finished prepare() results stay cached for history-to-resume reuse — it never prevents a cold prepare from fully materializing a session. So lowering it to 1 only stops *re-*preparing the same session back-to-back; the first prepare of each of the 15 sessions still pays the full decode+clone price. Your ~2.7 GB vs ~3 GB delta is exactly the reuse-cache saving. This knob can't bound boot cost — a per-prepare bound has to live inside prepareCore.

2. The heap amplification is a decode+clone cascade, not raw log size.
One cold prepare runs, in sequence: loadStored → readPrefix → readZstdPrefix decodes every frame / scanLog parses the complete buffer (session-persistence-jsonl/src/index.ts:209-215, 310-322); then prepareCore → adoptStoredEvents migrates + deep-freezes every event (coordinator.ts:560-573, 905-930); then Session.create/fromRestore + snapshotSessionEvent = adoptSessionEvent(structuredClone(event)) per event (core/session/src/index.ts:192-194). So each event is deep-cloned ≥2–3× in one prepare, plus the freeze walk. 15 × (1.8–2.7 MB compressed ≈ 20–40 MB decoded) × clone amplification ≈ your ~200 MB/session and the structuredClone ~60% CPU share — consistent.

3. On current master I could NOT find a boot sequence that enumerates all persisted sessions.
Traced every boot-adjacent path: persistence.list() is header-only by design ("Read only headers so listing scales with session count, not log size", session-persistence-jsonl:446-449); the workspace registry's [Service.init] uses that header-only list; ensureRootEncoding is exists() probes only; the persistence coordinator's start iterates live sessions only (coordinator.ts:1136); the in-process client boot (startInitialSelection) opens exactly one session (the current/recent workspace's); and the session.list cold blank probe is capped at DEFAULT_COLD_BLANK_PROBE_MAX_BYTES = 1024 bytes. So the all-15-materialized heap you measured must be demand-driven — whatever surface opens sessions after boot (client restore of the current session, a reconnect, an opened conversation) pays the full per-session decode+clone price, and each additional opened session is another ~200 MB cliff. One data point that would pin the trigger: which build/version were you profiling? Your frame names/line numbers (readSession :811, consumeEventLine :275, snapshotSessionEvent :1172) don't match current master (those names don't exist / live at different lines here), and master has since landed 2be3e12965 "harden cold session metadata probing" (Aug 13) — so a re-run on current master would tell us whether part of the boot trigger was already addressed.

Fix direction this adds: bound prepareCore itself — decode+adopt a bounded window (mirroring the readFrom suffix path) so N cold opens cost O(window) each instead of O(full log), and make the first render lazy. Combined with the already-listed items (bounded readPrefix, listArtifacts skip+warn, dsh session repair), the corruption family + the scale family then share one bounded-read core.

[wellorbetter]

I prepared a tested reference hardening patch in a fork so maintainers (or a coding agent) can inspect or cherry-pick it directly:

• Branch: https://github.com/wellorbetter/deepseek-harness/tree/fix/session-history-responsiveness
• Commit: wellorbetter@159c5e4
• Compare: wellorbetter/deepseek-harness@master...fix/session-history-responsiveness

Reproduction and evidence

The observed sequence was:

1. dsh web printed http://127.0.0.1:3080 and the HTTP listener existed.
2. Opening or typing into a previously large Session made the UI appear globally unavailable.
3. Some history rows did not load.
4. One affected JSONL artifact contained a backward sequence conflict produced by two competing branches; another valid artifact contained 444,316 logical events across 21,153 Zstd frames before lossless repacking.
5. After backup-first offline repair/repacking, both historical Sessions loaded again, which isolates the incident to persistence concurrency plus cold-read behavior rather than the HTTP bind itself.

Source-level root cause

There are three interacting causes:

1. PersistenceCoordinator serializes writes only inside one backend instance. The JSONL README documented the one-live-writer requirement, but the backend did not enforce it across backend instances/processes. Two processes could therefore resume one Session with independent cursors and append conflicting seq values to the same append-only file.
2. The scanner correctly fails loud on duplicate/backward/gapped sequence numbers. That is safer than guessing which branch owns tool outcomes, but it means an already-conflicted Session cannot load normally.
3. Cold Zstd reads materialize and scan the whole history. Master already yields at frame boundaries, but the scheduling slice was 500 ms, and unchanged corrupt artifacts were rescanned on every retry. A large automatically selected history could therefore monopolize the Web host long enough to make the whole UI look down even though port 3080 was listening.

Fix in the fork

The patch adds:

• A hashed, per-Session cross-process writer lease. Complete owner metadata (PID, host, time, random token) is atomically published without replacement before append/repair. A competing live writer fails before mutating the log; a dead same-host owner is reclaimed safely; unrelated Session ids remain independently writable; leases release only after coordinator quiescence.
• A 16 ms Zstd scheduling slice instead of 500 ms, retaining frame-boundary cancellation while keeping the Web event loop responsive.
• A bounded, revision-scoped cache for deterministic parse/decode failures. An unchanged corrupt artifact fails immediately on repeated UI requests; repairing/replacing the file changes its stat revision and automatically triggers a fresh read.
• Regression coverage for contention, release/takeover, stale-owner reclamation, revision invalidation, and the shorter scheduling slice.

Validation on Windows/Node 24:

• JSONL targeted suite: 235/235 tests passed.
• TypeScript project build passed.
• Oxlint passed.
• Agent Note, bilingual pairing, Markdown wrapping, README limitation/model-experience, and exported-JSDoc gates passed.

This is intentionally an interim hardening patch, not the complete bounded-window/paginated history design discussed above. It prevents the corruption recurrence demonstrated here and removes the worst repeated-scan/event-loop amplification; a true O(window) prepareCore/history path remains valuable follow-up work.

Existing corrupt histories

The runtime patch intentionally does not silently rewrite an already-conflicted log because a backward seq creates two plausible branches and automatic selection can hide a tool outcome. Recovery should remain backup-first: stop every DSH writer, preserve the original artifact, produce a separate repair candidate, verify contiguous logical events and official inspect(), then replace only after the candidate passes. In my reproduction, that process restored both affected histories.

Until an upstream fix lands, the simplest safe workaround is: never let two DSH processes write the same Session, stop all other DSH instances before resuming it, and back up the persistence root before repairing any existing conflict.

[argszero]

Reviewed the patch (fork branch fix/session-history-responsiveness, commit 159c5e4f6) against master and ran its test surface — all 3 touched spec files pass (writer-lease 2/2, jsonl 152/152, zstd 81/81). Three mechanisms, and here is how they map onto the corruption family this thread tracks:

1. Cross-process writer lease (writer-lease.ts + ensureWriterLease) — fixes the #1333/#1452 mechanism.
Atomic link()-based lock per session under <root>/.dsh-jsonl-writer-locks/, claimed at the top of appendBatch (covers both the appendLines and materialize branches) and in repair, released by close() after the coordinator drains. Same-host stale locks are reclaimed via PID liveness; foreign-host locks fail closed. This is the first implementation of the family's cross-process writer fix — it addresses exactly the open('a')-without-lock + per-instance seq = log.length duplication that #1333/#1452 documented. The stale-takeover design is sound (full metadata written+fsync'd into a 0600 candidate, atomically published, then rename-based takeover that only one contender can win; rm on release is token-guarded so an externally replaced lock is never unlinked).

2. Cached deterministic read failures — kills the repetition of the stall.
readFailures is keyed by path and validated against the artifact's revision (stat before reuse), bounded by preparedSessionCacheSize. After one failed decode, every subsequent read of the same unchanged artifact throws instantly instead of re-decoding 21k frames — that's the GET / retry loop that piled up CLOSE_WAIT in the original report. Reusing preparedSessionCacheSize as the bound is a good fit for its retention semantics.

3. ZSTD_DECODE_YIELD_INTERVAL_MS 500 → 16 — makes the unavoidable full decode interruptible. The server can interleave requests during a big decode instead of pinning the main thread for the whole file.

What the patch does NOT cover (for the maintainer's cherry-pick decision)

• First cold open of a valid large session still fully materializes — no bounded/seekable read. 7889545's 15-valid-sessions scale case: each session's first open still pays the full decode+clone (~200 MB heap each; snapshotSessionEvent's structuredClone is outside the yield path, so the ~60% CPU share and the multi-GB ceiling remain). This patch fixes stall repetition and stall interleaving, not materialization size. The bounded prepareCore/readFrom-suffix design is still the complementary fix.
• The [Bug] A single corrupted session log prevents the whole workspace from booting (loader entry "workspace" fails to apply) #1473 boot-block path is untouched: a corrupt artifact still fails the listArtifacts first-frame header read (assertZstdHeaderFrame), so workspace startup can still refuse to boot. The cached-failure only helps reads routed through readPrefix, not the header listing. The quarantine/list-time-skip direction (Add a session doctor and versioned migrations #1518, dsh-doctor S11) remains separate.
• The Bug: corrupt session log after unclean exit — replayed events cause seq gap in committed region, history unrecoverable #1497 replay mechanism is orthogonal: the lease prevents future concurrent writers, but a process that crashed mid-write-behind can still re-submit from a stale in-memory cursor on resume. That needs the persisted-watermark fix.

Net: this is a solid, well-tested PR-ready hardening that closes the concurrency gap and the repeat-stall gap; the pure-scale materialization and the boot-block paths need the two follow-ups above. Worth opening as a PR when the upstream PR channel reopens.

[moonquake2004]

7889545's scale numbers are directly relevant to the offline side: their 15 sessions (each ~20–40MB decoded) cost ~200MB heap apiece — but that's below our current per-session oversized threshold (decoded >10MB / >10k frames), so our whole-session scan would NOT have flagged the exact profile that stalled their boot. The lesson: the materialization-risk bar belongs on the workspace total, not per-session — boot/heap cost is cumulative across adopted sessions.

Done — S11 now carries a total-materialization line: it sums decoded bytes/events across all sessions and flags when the workspace total crosses a bound (150MB decoded). Verified: a synthetic 5-session / ~190MB-total profile now trips it even though each session is individually ordinary; our own live profile (~14MB largest) stays clean. So the 7889545 profile shape is now caught offline, before boot.

Also agree with the patch review's remaining-gaps framing: the writer-lease / cached-read-failure / 16ms-yield patch closes the concurrency and repetition gaps; the first-open materialization, #1473 boot-block, and #1497 watermark follow-ups are exactly what S11 surfaces offline (corrupt → quarantine, oversized → warn).

[moonquake2004]

Following up on the offline-side threshold — I re-calibrated S11's materialization-risk metric with real measurements instead of a heuristic, and want to share both the method and the residual uncertainty:

Measurement (real, our 0.1.0-rc.6 web session, 420k small events / 15MB decoded):

• Actual object-graph heap for parse + expandRow-style materialization: 103.7MB (~259B/event) — measured in a child process replicating the expansion, not estimated.
• That's before the harness's deep-freeze + structuredClone (2-3× per argszero's trace), so real harness cost ≈ 300MB+ for this one session.

Recalibrated formula:

estHeap = max(events × 600B, decodedBytes × 6)   # 600B ≈ 259B/event × ~2.3 clone; ×6 for byte-dominated (conservative middle of your 5-10×)
warning line: 1GiB (configurable via DSH_DOCTOR_HEAP_MB)

The max() covers both regimes: small-event sessions (events term) and large-event sessions (bytes term).

Scenario matrix (verified with synthetic profiles): events-term triggers / bytes-term triggers / below-line clean / 7889545-scale (176MB decoded) trips the default 1GiB / corrupt → quarantine / plain-.jsonl sessions now scanned too / boundary exact (1.6M events ≈ 960MiB → no total-line; 1.8M events ≈ 1030MiB → trips).

Residual uncertainty (worth your input): the clone factor 2-3× is from source analysis, not measured in the harness — a real heap number for a known session set (7889545's 3GB on 15 sessions implies ~6-10× on bytes) would pin the byte coefficient; the exact stall point per RAM/OS is still unmeasured. The per-session threshold (10MB / 10k frames) stays as an independent safety net.

Also fixed while auditing: plain .jsonl sessions are now scanned (previously only .zstd), and a bug where plaintext files were mis-flagged as corrupt.