Unbounded session_projcache growth stalled the event loop on 0.1.1-rc.2 — incident evidence + bounded-cache follow-up

[HiSeax]

Unbounded session_projcache growth stalled the event loop for 30–120s on 0.1.1-rc.2 — incident evidence + bounded-cache follow-up

TL;DR: On 0.1.1-rc.2 the session_projcache.json unit is one global document rewritten in full on every storage write. With no eviction anywhere, it grew to 64.6 MB (1,170 session slices) on our machine, and every turn/end checkpoint write blocked the single-threaded event loop long enough that the health check timed out and our watchdog force-killed the server — 8 times in one night, each kill losing in-flight tool calls. The per-record layout in 0.1.2-alpha.1 fixes the giant-file rewrite (thank you!). The remaining gap is that the cache is still unbounded — records accumulate forever. This post collects the incident evidence and proposes a small bounded-cache follow-up; we are happy to open a PR if that is welcome.

Incident evidence (0.1.1-rc.2, Windows, default json backend)

• Kill timeline (watchdog log): 00:37:24, 00:38:47, 00:41:27, 00:49:19, 00:58:20, 01:07:43, 01:13:10, 01:36:47 (local time, one night). Each kill followed a turn/end in an active 6.4 MB session by 20–112 s (exact turn/end timestamps extracted from the session JSONL).
• Smoking gun: an interrupted atomic-rewrite temp file .4338d4e4….tmp (63.53 MB) left in the storages root with mtime exactly equal to one of those turn/end timestamps (00:57:56) — the checkpoint write was mid-rewrite of the 63.5 MB document when the process was killed.
• Mechanism (source, 0.1.1-rc.2):
  • @deepseek-ai/dsh-storage-json — serialize() JSON.stringifies the whole unit and writeAtomic() rewrites the whole file per write primitive ("every write primitive mutates it and republishes the whole file atomically").
  • @deepseek-ai/dsh-session-projection-cache — Config has only writeEveryEvents / writeIntervalMs; turn/end and disposal are mandatory write points; no cap, no prune API.
  • @deepseek-ai/dsh-storage-domain — KvTable.delete() exists but the cache service never exposes it.
• Consequence: synchronous JSON.stringify of a 64 MB document + fsync on the turn/end path blocked the loop for 30–120 s; health probes (local HTTP and an SSH reverse tunnel) failed together; the watchdog killed a healthy-but-busy process.
• A 2-minute stall later caught with CPU sampling (82% busy) was not killed only because we patched the watchdog to defer recovery when the server is busy — that stall also originated from checkpoint writes.

Remaining gap in 0.1.2-alpha.1

The per-record layout removes the single-giant-document rewrite, but the cache is still unbounded: one document per session accumulates forever (our 1,170 slices → 1,170 files), with no LRU and no way to reclaim records for retired sessions. Growth is slower but still monotonic, and listing/scanning cost grows with it.

Proposal (small, opt-in): add maxRecords (0 = unbounded, previous behavior) to session-projection-cache config and a prune() pass after writes: evict oldest records by identity.createdAt, never evicting sessions currently live in the registry. Per-record layout makes each eviction a single small delete — cheap and safe (eviction only lengthens the next cold read).

Reference implementation

We ship this exact policy as an external patch plugin for 0.1.1-rc.2, including the full evidence chain:

• Repo: https://github.com/HiSeax/dsh-projcache-lru (EVIDENCE.md contains the timeline, artifacts, and source-level citations)
• Behavior: 60 s sweep, 8 MB soft / 10 MB hard caps, eviction order orphan → subagent (delegationDepth > 0) → oldest main session, live-session exemption, small delete batches.

Happy to send a PR for the maxRecords + prune() follow-up if maintainers are interested.