UI lag during token streaming / many background subagents: measured root cause (per-token fan-out, no stream batching)

[silvertakana]

Summary

The dsh Web GUI (127.0.0.1:3080) becomes visibly laggy while tokens stream and/or many background subagents run, even though the machine looks idle. Live profiling on both sides shows the same root cause: per-token event fan-out with no server-side stream batching, plus O(surface) client work per message. Fix direction: server-side stream coalescing, client-side snapshot memoization, and cheaper token metering.

Environment

Item	Value
dsh	rc.6 (npm, published 2026-08-13)
OS / browser	Windows 11, Chrome-based GUI
Permission mode	danger-full-access (all sessions)
Default model	deepseek-v4-flash via local router (stated for completeness; irrelevant to streaming smoothness)

Reproduction conditions

• Tokens streaming in the foreground session while several background subagents run
• Typing, scrolling, and UI updates feel delayed; machine load looks normal throughout

Server-side evidence

Live V8 CPU profile via CDP over the existing --inspect port (no restart needed):

Metric	Value
Node process CPU	~55% of one core continuously; avg 1.095 CPU-sec/2s window over 80s, peaks 89%
Samples in FrameQueue.iterate	63% of non-idle samples (3,249 of ~5,176)
Hot site	dsh-host-apiproxy/lib/index.js:1108 (host-to-client event mux drain loop)

Mechanism:

• events.mux (dsh-host-apiproxy/lib/index.js:3525) creates one queue per connected browser, subscribes to every session, and pushes one frame per session/event (index.js:3556+)
• Every assistant/chunk synchronously fans out (dsh-session/lib/index.js:1444-1484) through all session-projection units, the token meter (dsh-token-meter/lib/index.js:500-522; structuredClone + deepFreeze of the full surface on measure()), persistence write-behind (structuredClone, 200ms batched zstd + fsync, dsh-session-persistence/lib/index.js:324-333,436), and one WebSocket frame to the browser
• With N subagents streaming, this multiplies by N on a single event loop
• Client-side microtask batching (AbstractApiClient.onEnvelope, dsh-host-apiproxy/lib/index.js:5210-5253) covers only diagnostics listeners, not the conversation renderer

Browser-side evidence

Chrome DevTools performance trace (38.9s window, 564k events):

Metric	Value
Renderer main thread busy	99.04% (38,520.8 ms of 38.893 s; idle 0.96%)
Per-second busy	882-1043 ms in all 39 one-second buckets (starvation, not bursts)
Tasks > 50 ms	Only 3 (max 101.7 ms)
WebSocketReceive	13,201 messages at 339.4/s; 99.1% immediately followed by a publish FunctionCall
Hot sampled JS (122,992 samples)	buildListSnapshot 5,305 ms self (dsh-client-runtime/client.js:8552); SessionManager.notifier 4,532 ms (client.js:7843); plus walk / flattenLineage / projectList / indexSubagentDescendants (37.5% of all sampled JS in dsh-client-runtime)
Layout + Paint	~660 ms (1.7% of window, ~0.5 ms/paint) - not the bottleneck
GC	1,230 ms (1.5%), max pause 9.9 ms - not the bottleneck
Heap	Sawtooths 114.5 -> 253.9 -> 169.3 MB, net +19.5 MB - churn from per-message snapshotting, not a leak

Verdict: two-sided, same root cause. (a) The server pushes ~339 frames/s with no batching; (b) the renderer does heavy per-message snapshot/projection JS. "Too much layout/paint per frame" is not supported (1.7%).

What was ruled out

• Model router (upstream): affects time-to-first-token, not streaming smoothness
• Machine load: event loop idle ~70% of profile time
• Plugin timers: deadman tick touches only armed switches; compaction is event-driven; session-query-sqlite is lazy, heavy only on explicit history search
• Memory leak: heap sawtooths, net flat
• Distinct from the known Windows "wedge" incident (ACL propagation on workspace-write sessions causing total HTTP blackout): this is constant lag on danger-full-access sessions, no blackout

Proposed fix direction

1. Server-side stream batching/coalescing in dsh-host-apiproxy: flush the conversation stream at most ~every 50-100ms instead of one frame per token. Cuts ws msg/s and publish count ~10-50x; biggest lever
2. Client-side memoization/debouncing of buildListSnapshot / projections in dsh-client-runtime
3. Stop the token meter's per-chunk full-surface copy; measure on message boundaries
4. Throttle background-subagent pushes while the foreground session is read

Note

A local workaround (server batching + client debounce) is in progress; will report back with before/after measurements once applied and verified.

[silvertakana]

Post-restart verification: both fixes are live and the lag is gone.

Applied as local patches to the installed packages (dsh-host-apiproxy + dsh-client-runtime, both 0.1.1-rc.2), then restarted the web process (patched files loaded, confirmed by file mtime vs process start).

Server side (V8 CDP profile, 25s while tokens streamed, 15,682 samples):

• BEFORE: FrameQueue.iterate at dsh-host-apiproxy/lib/index.js:1108 was 63% of non-idle samples; the dsh main process burned ~1.095 CPU-sec/2s (~55% of one core) continuously.
• AFTER: FrameQueue.iterate is COMPLETELY ABSENT from the profile. dsh-host-apiproxy total is 78 samples (~0.5%). Top self-time entries are now natural streaming work: get events (dsh-session/lib/index.js:1401) 2297, GC 961, writev 603, cordis get 672.

Browser side (renderer main-thread, active GLM 5.3 streaming):

• BEFORE: main thread 99.04% busy for the entire 38.9s window (idle 0.96%), 339 WS messages/s, every second in the 882-1043ms busy range. That starvation caps rAF well below 60fps.
• AFTER: rAF frame-rate probe during active streaming = 235 fps (1178 frames / 5.0s). The renderer is not pegged; the UI is fully responsive while tokens stream.

What changed:

1. Server: FrameQueue.iterate now coalesces all buffered frames into one batch per yield (nested drain-then-await), instead of yielding one frame per queue item. Both consumers (events.mux, events.host) flatten batches back to individual frames at the boundary, so the wire format is unchanged. Streamed tokens now cost one drain + one serialization pass per batch, not per token.
2. Client: stream-driven list rebuilds (projectionStore.subscribeAny, session/projection) are now debounced to requestAnimationFrame cadence via Notifier.markFrameDirty (at most once per frame) instead of a synchronous rebuild per WS message. User-initiated paths (selection, subscribe buttons, etc.) remain synchronous.

Measurement notes: WS msg/s is deliberately NOT the success metric here; downstream consumers re-flatten batches to one frame per WS message by design, so msg/s stays the same while per-message work drops. The meaningful metrics are server CPU share and renderer main-thread availability, both shown above. Playwright MCP CDP tracing returned no events in this environment, so renderer health was measured with a rAF frame-rate probe instead.

Happy to prepare this as a proper upstream PR if the maintainers want it.

[Vladimir-Human]

Measured on 0.1.2-alpha.4 (commit 4e84901e, Windows, web profile): background-session streams saturate every connected client's main thread — and a client-side throttle fixes it

Independent measurements confirming the fan-out cost described in this discussion, plus one twist that makes it worse in multi-agent setups.

Setup. One background agent streaming continuously (a subagent generating a long LLM answer; compressed journal growth ~550–830 B/s ≈ 15–25 events/s). A separate Chrome tab connected to the web UI, sitting on the empty workspace screen (no session open, DevTools closed). Renderer CPU measured process-level via TotalProcessorTime deltas — no injection, no DevTools overhead.

Measurements

• Tab connected to the host where the agent streams: renderer main thread ~101% CPU (97–106 over a 90 s window), on the empty workspace screen.
• Same tab, same build, same DOM, connected to an idle instance of the same build (shared session root): 0.2% CPU (max 1.6%).
• Owner's daily tab over 45 min with agents running: 99.3% avg (532/539 samples > 50%).
• With two concurrent streams (own answer + a background agent), the visible text froze for up to 13.75 s while the host journal stayed continuous and host CPU was < 10% — the starvation is purely client-side.

Mechanism. Every delta of every session — including sessions the user is not viewing — is fanned out to all connected clients, and the client's coarse notification channel (the one feeding the session-list/manager projection) triggers a re-render per event at frame rate. We first enabled alpha.4's built-in rAF batching for the hot stream paths (projection.changed / subscribeAny / append → markFrameDirty), which removed the constant 99% jank for a single visible stream — but background streams alone still saturate the main thread, because the coarse channel keeps firing per event at frame rate.

Client-side mitigation we shipped locally (measured). Throttled the coarse channel's publish interval to 333 ms (3 updates/s), keeping the visible session's own channel at full frame rate. Result with the same background stream running: renderer CPU 0.5% avg (max 8.6%); visible-stream smoothness unchanged (median DOM update interval 100 ms, p90 104 ms; max plateau 4 s = the model's hidden reasoning phase).

Suggestion. Server-side batching as proposed here fixes the fan-out cost for all clients at once — we would rather delete our local patch. An additional client-side safeguard worth considering: throttle or de-prioritize notifications for sessions that are not currently rendered, so UI cost stays independent of how many agents stream in parallel.