[Bug] Eager projection-cache load dominates dsh web startup: 969 records / 209 MB → 5.3 s of validation (26%)

[hjwsm1989]

Summary

dsh web startup time grows with the number of persisted sessions. On a long-lived
installation (968 session directories, 3+ months of use) startup went from 12 s to 25 s in
three days, and RSS right after boot is 1.42 GB versus 454 MB with an empty data directory.

The dominant term is the eager, unconditional load of the entire session projection cache —
including a recursive isJsonValue validation of every value on the read path — for data
the process itself just wrote.

Environment

• dsh 0.2.1-rc.2, Linux x86_64, Node 22.22.0
• profile web: 199 plugin entries (190 from @deepseek-ai/dsh-base + @deepseek-ai/dsh-web-app, 9 third party)
• data directory: 968 session directories, 798 MB session logs;
  storages/session_projcache/sessions/ = 969 JSON files, 209.1 MB (avg 215 KB, max 2.9 MB, avg 25.8 rows per record)
• startup measured from process start to the dsh web: http://… ready line

Reproduction

1. Use a data directory with ~950 sessions and their projection-cache records (~200 MB).
2. Run dsh web --no-open and time until the ready line.
3. Control runs on isolated copies (cp -a --reflink=auto) of the same directory:

data set	time to ready	RSS
full (969 records, 209 MB)	15.9 s	1420 MB
records older than 14 days deleted (620 files, 105 MB)	13.3 s	949 MB
empty data dir (no sessions)	7.4 s	454 MB

Reading and parsing the cache directory alone (readdir + readFile + JSON.parse, no
validation) takes 2.4 s; readdir itself is 0.8 ms, so it is all parse work.

Current behavior

CPU profile of one boot (20.3 s wall, 18402 samples):

• parseRecord subtree (including zod valueSchema.parse): 5327 ms (26%)
• ESM compileSourceTextModule: 1925 ms
• GC: 1430 ms
• dsh-util-values self time: 3849 ms — walkJsonValue 1462, hasIntrinsicConstructor 895,
  enumerableStringKeys 638, isJsonValue 504

The chain:

• dsh-session-projection-cache/lib/index.js:151 —
  const domain = await this.ctx.storageDomain.open(projectionCacheDomainSpec);
• dsh-storage-domain/lib/index.js:355 async open(spec) → :364
const snapshot = await unit.loadAll(); — unconditional, no notion of "which records does
  the boot path actually need".
• dsh-storage-json/lib/index.js:325 loadPerRecordState() — readdir then
  Promise.all(readRecord) over every *.json (:410-426), each one JSON.parse
  followed by tableSpec.valueSchema.parse.
• dsh-session-projection-cache/lib/index.js:28-32 —
  checkpointRow = z.object({ ver, seq, val: z.custom(isJsonValue) }), so every row's val
  is walked recursively (25.8 rows per record on average, up to hundreds for long sessions).

Two aggravating factors:

1. Asymmetric validation. The write path is cheap — put() only does
   snapshotJsonValue(rows) (lib/index.js:348-349) — while the read path re-validates every
   value deeply on every boot. We pay the full cost for data that no other process can have
   corrupted.
2. Pretty-printed records. Records are serialized with
   JSON.stringify({ version, record }, null, 2), so 209 MB on disk where compact JSON would
   be 107 MB: 2× the bytes to read, decode and validate, for no benefit.

dsh-session-projection-cache/README.md already documents "No eviction or retention
surface — records accumulate per session", so for a long-lived installation this cost is
monotonically increasing (this one adds 10–20 sessions per day).

Related context: the rest of the boot path is cheap and linear — session/list costs ~323 ms
for 968 sessions (one zstd header read + stat per artifact). The projection cache is
~100% of the startup regression and ~65% of post-boot RSS.

Expected behavior

• open() should be able to open a domain without reading every record. A lazy
  get(key) / load(key) path would let plugins that only touch a few records avoid the
  full snapshot.
• Validation belongs on the write path (or behind a debug flag), not on every read of
  self-written data.
• Records should be persisted as compact JSON (or zstd), not pretty-printed.

Patch sketch

• storage-domain: open(spec) accepts { eager?: boolean }; expose get(key); keep
  loadAll() for callers that genuinely need a full snapshot.
• session-projection-cache: do not open() in Service.init; open lazily on the first
  projection read, or open with eager: false and fetch per session.
• storage-json: move the isJsonValue check into put() / serializeRecord, or gate it
  behind a --validate-records flag.

Happy to run any patch against this data directory and report boot time and profile
before/after — the full .cpuprofile is available if it helps.

[hjwsm1989]

Additional browser-side measurements from the same DSH 0.2.1-rc.2 deployment. This is a separate bottleneck from the server startup profile in the original post.

• On a browser reload, POST /api/session/list returned 994 sessions. With nginx gzip enabled, the response was 9,540,875 bytes decoded / 1,359,683 bytes transferred. Chrome measured 13.19 s total: 2.90 s waiting for the response and 10.10 s receiving it. The waiting phase includes network effects and should not be interpreted as server CPU time.
• In a captured response of comparable size (9,540,409 bytes), contextHeaders accounted for 6,279,015 bytes and turnOutline for 1,477,028 bytes. Together these projection values were about 81% of the list JSON.
• After I backed up and removed some archived sessions, and separately removed a third-party Agent Teams plugin, a reload returned 6,974,246 bytes decoded / 1,096,304 bytes transferred and session/list took 6.13 s. Both changes happened between samples, so this is a correlation, not an A/B attribution to either change.

The original post's ~323 ms session/list measurement is server-side list work during startup; the browser measurement above includes response delivery over a VPN. The list payload size appears to be a significant part of the page-open delay.

Could the list API send only fields needed for sidebar rows, fetch heavier projections when a session is opened, and support effective pagination/incremental loading (including archived sessions)? In 0.2.1-rc.5, SessionListRequest declares a cursor, but the list implementation ignores the request and returns all visible rows.

[hjwsm1989]

Runtime supplement: the same projection data also stalls the page open, and the list call serializes the session open.

Same deployment (dsh 0.2.1-rc.5 now, data dir at 881 cache records / 186 MB, 287 session directories), continuing from the browser-side comment above. This adds two things that comment could not separate: server time vs delivery time, and what the stalled renderer does to the session that is being opened.

I added rt=$request_time urt=$upstream_response_time to nginx, so the two phases are now measurable separately. One page load:

POST /api/session/list   1,128,369 B    rt = 11,578 ms    urt = 2,961 ms    delivery = 8,617 ms

• urt = the host building the list: 2.96 s for 1.13 MB. This is server work, not network.
• The remaining 8.6 s is the body reaching the renderer.

The list call serializes the session open. DevTools' Socket panel (/api/remote.mux) for the same window shows session/follow opened at 11:00:46 and its snapshot frame (1,113,657 B) delivered at 11:00:54 — 8.8 s later, while session/list was still in flight. Other requests in that window (/plugins/dsh-agent-teams/state, /api/schedule/catalog, /plugins/agy-link/status) answered in 5–100 ms, so the host was not globally blocked: the renderer was busy with the list payload, and both the list body and the WebSocket frame queued behind it. Opening a session is therefore gated on the list response, not only on that session's data.

The snapshot has a message cap, not a byte cap. follow() in dsh-api-session-controller uses paginate(events, undefined, request.maxMessages ?? DEFAULT_MAX_MESSAGES, …) with DEFAULT_MAX_MESSAGES = 50, yet this session's snapshot was still 1.1 MB because single messages (tool results) run to hundreds of KB. A byte budget would behave better than a message count here; 50 messages is not a bound on bytes.

One contributor was a third-party plugin, and it is now fixed. The Agent Teams activity panel polled /plugins/dsh-agent-teams/state every second and received every team of every registered workspace: 910,084 B per response on this installation, while the panel only ever renders the current session's teams. dsh-agent-teams #330 (PR NanmiCoder/dsh-agent-teams#332) scopes the request to the current session, adds a summary projection, ETag/304, a hidden-tab cadence and a per-file identity cache. The same poll now costs 43 B or 304, 11–15 ms, and an unchanged poll no longer wakes the renderer. Worth separating when attributing the remaining page-open cost: the plugin poll was real (910 KB/s-class on a long-lived install) but independent of the projection-cache problem.

Suggestion for the list API, on top of the previous comment:

• Return sidebar-row fields only (sessionId, title, updatedAt, running, blank, cwd, parentSessionId, formatStatus) and fetch projections.values lazily for rows that are visible or opened.
• If that is too large a change: keep the shape but cap projections.values by bytes, and drop the blocks a sidebar never reads (contextHeaders, turnOutline, contextBreakdown, subagentCatalog, userQuestions).
• Give session/follow a byte budget in addition to maxMessages.

The startup problem in the original post and this runtime problem are the same underlying data (very large projection values, eagerly materialized); the startup patch reduces boot cost but does not reduce what every page load has to ship.

[hjwsm1989]

Which projection blocks actually cost the bytes: per-key shares and a real A/B.

I reproduced the list RPC against the live host so the numbers below come from the real response rather than an estimate:

POST /api/session/list
{"type":"client-request","rpcId":"1","method":"session/list",
 "payload":{"args":{"_request":{}}}}

Result: 880 items, 7,419,220 B decoded, 890,205 B gzip(-6). nginx reports 1,128,369 B transferred for the same request at its gzip_comp_level 1, so roughly 1.1 MB on the wire per page load.

projections.values is 6,936,508 B = 93.5% of the decoded response. Everything else the list returns (ids, titles, cwd, running flags, timestamps) is under 7%.

Per key, aggregated across the 880 rows:

projection key	rows	decoded	share	gzip(6) of that key	share of gzip
contextHeaders	631	4,597,898	62.0%	56,656	8.5%
turnOutline	639	1,269,614	17.1%	458,255	68.7%
contextActivity	631	231,923	3.1%	40,741	6.1%
imageLimits	627	125,400	1.7%	655	0.1%
modelSelection	627	89,891	1.2%	1,419	0.2%
sessionStats	639	77,669	1.0%	17,572	2.6%
contextTimeline	50	75,823	1.0%	15,667	2.4%
tokenUsage	639	62,345	0.8%	8,704	1.3%
subagentCatalog	639	59,974	0.8%	19,671	3.0%
todos	639	45,390	0.6%	13,230	2.0%
contextPressure	639	44,661	0.6%	5,725	0.9%
contextBreakdown	639	40,110	0.5%	4,174	0.6%

The two halves disagree, which matters for choosing a fix:

• contextHeaders is 62% of decoded but only 8.5% of compressed. It is extremely repetitive (the same tool-name/header structure repeated per request), so gzip eats it. Removing it barely helps the user waiting on the wire, but it does help parse time and resident memory.
• turnOutline is 17% of decoded but 68.7% of compressed. It is the dominant transfer cost, and it is the one users wait for.

Real A/B on the same captured response (rebuild the document with one key removed, then JSON.stringify + gzip -6):

simulation	decoded	Δ decoded	gzip	Δ gzip
baseline (all 880 rows)	7,419,220	—	890,205	—
drop turnOutline	6,113,367	−1,305,853	325,066	−565,139 (−63.5%)
drop contextHeaders	2,783,310	−4,635,910	717,612	−172,593 (−19.4%)
drop both	1,504,111	−5,915,109	209,955	−680,250 (−76.4%)
drop top-6 keys	936,324	−6,482,896	132,527	−757,678 (−85.1%)
drop all projections.values	215,443	−7,203,777	30,407	−859,798 (−96.6%)

If the goal is "the sidebar opens faster", dropping turnOutline alone removes almost two thirds of the transferred bytes. If the goal is boot cost and RSS (the original post), contextHeaders is the one that dominates decoded size.

Archived sessions are a large share of the list. On this data directory 182 of the 880 rows are sessions the user has archived, but they are the oldest and heaviest ones:

subset	rows	decoded	share	gzip	share
all rows	880	7,419,220	100%	890,205	100%
archived rows removed	698	4,844,531	−34.7%	428,123	−51.9%

So half of the transferred list is content for sessions the sidebar is not showing by default. Excluding archived sessions from the default list (or paginating), and fetching their projections only when the archive view is opened, is a cheap and independent win — smaller than the turnOutline change but it needs no API redesign.

Suggested priority, given these numbers:

1. Do not ship turnOutline in the list payload (biggest wire win, ~64%).
2. Do not ship archived sessions' projections in the default list (~52% of the remaining wire cost).
3. Only then consider the broader "sidebar fields only / lazy projections" redesign from the earlier comment.

[hjwsm1989]

Follow-up after reducing the session set: two remaining page-open paths

I repeated the browser test after the two large background costs had been removed:

• archived sessions were backed up/removed, leaving 118 rows in session/list (103 user sessions + 15 active team-member sessions);
• the Agent Teams state poll is now the scoped summary/ETag implementation from perf: scope, summarize and cache the activity state poll (#330) NanmiCoder/dsh-agent-teams#332.

The new session/list response is 904,144 B decoded / 122,180 B gzip, and the host builds it in about 0.38 s. Startup is about 7 s with ~453 MB RSS. This confirms that the original projection-cache/list problem was real and that reducing it works, but some individual sessions still take 8–13 s to become usable. I found two distinct remaining paths.

1. Attachment-heavy sessions: server TTFB is fast; transfer and cache policy dominate

One older session contains four PNG files served through /api/file?path=.... Opening it fetched about 1.1 MB in total; each image completed in 8.23–8.72 s. A clean re-request of one 376,249 B PNG measured:

• total: 9.60 s
• connection: 135 ms
• waiting for server response: 58 ms
• content download: 9.38 s

The response contains:

Cache-Control: private, no-store

So this case is not slow server computation. It is slow delivery over the VPN, compounded by DSH forcing Chrome to download unchanged attachments again on every open. Revalidation (ETag/Last-Modified + private, max-age=0, must-revalidate), content-addressed URLs with a private max-age, lazy image loading, and thumbnails would make repeat opens much cheaper. The text UI should also become usable without waiting for every attachment to finish.

2. Large text/history sessions: the list is no longer the gate

A different session with 2 rounds / 69 steps / 7.9M tokens stayed on “loading history” for about 12.7 s after the click. During that interval the ordinary Fetch/XHR traffic was only 13.1 KB (mostly small status requests and 304 Agent Teams polls). This points to the existing /api/remote.mux / session/follow path, snapshot replay, client parsing, and rendering rather than session/list.

This supports the earlier suggestion to cap session/follow by serialized bytes, not only by 50 messages. A single tool result or attachment-bearing message can still make the initial snapshot very large. Incremental replay and progressive rendering would also prevent one large history from blocking the whole view.

While switching sessions I also captured this frontend error:

Sidebar Session opening failed: Error: Session reference \"session-…\" is released

The stack goes through ClientSessionReference.release and SidebarSessionView.dispose, which may indicate an open/dispose race when switching sessions and is worth checking separately.

Relationship to the original report: the startup and list regressions are both caused by eagerly materializing large per-session data across the whole retained set. After that set is reduced, the same design still appears at the single-session boundary through an unbounded-by-bytes follow snapshot. The attachment case is a separate transport/cache bug that amplifies the delay on VPN links.