[Bug] dsh web 100% CPU infinite loop in FrameQueue host->browser event mux #1262
Replies: 1 comment
|
Independent reproduction on Apple M3, with an open/close A/B test I can independently reproduce the same hot path on:
A/B observations
The A 3-second macOS Per-process I separately identified real CPU-heavy child workloads during the investigation, including short-lived Python checker processes. After those children finished and the Web UI was closed, Important caveatThis is not yet a clean stock-profile reproduction. The active web profile contains third-party plugins, including Five legacy Zstandard session logs had also been reframed to the current one-header-first-frame format before this reproduction. Their decompressed JSONL content was verified byte-for-byte identical before and after reframing, so no event content was rewritten. Despite that caveat, the open/close A/B result and the sampled If useful, I can run a clean-profile reproduction or an instrumented build that records frame type, |
Uh oh!
There was an error while loading. Please reload this page.
Environment
@deepseek-ai/dsh0.1.0-rc.6 (npx @deepseek-ai/dsh web);@deepseek-ai/dsh-host-apiproxy0.1.0-rc.6,@deepseek-ai/dsh-session0.1.0-rc.6dsh web, listening onhttp://127.0.0.1:3080Summary
Twice the
dsh webprocess pinned one core at 100% CPU (168 minutes, then 29 minutes) and the Web UI froze. Both samples show the same main-thread hot path:i.e. an async generator endlessly doing
queue.shift()→ yield → write to the local WebSocket, with a queue that never empties. The local127.0.0.1:3080→ browser connection carried 1.3 GB and 1.9 GB in two runs. The second occurrence had no external connection at all (only the local browser), so this is a local event-loop bug, not an LLM-API issue.Reproduction
Not 100% deterministic, but observed as follows:
dsh web, open the Web UI, run one or more sessions (agents/tools).(Reconnect/resume is the suspected trigger, but we could not yet isolate a minimal repro.)
Impact
Root cause analysis
The hot stack matches exactly one server-side construct: the
FrameQueueasync queue inpackages/host/apiproxy/src/...→ built tonode_modules/@deepseek-ai/dsh-host-apiproxy/lib/index.js:AsyncGeneratorResumeNext= the generator being resumed by thefor awaitconsumer;Builtins_ArrayShift → FastElementsAccessor::MoveElements → WriteBarrier= line 1173this.buffer.shift()(V8Array.prototype.shiftelement move + GC write barrier).yielded frame is written to the browser WebSocket by the consumer →node::StreamBase::Writev.iterate()only ever hits theawaitat line 1175 whenbuffer.length === 0. If a producer keepspush()ing, the buffer never empties and the generator spins synchronously (100% CPU) while emitting unbounded frames.Two streams consume this queue in the same file:
events.mux(line 3612): forwards per-session events to the browser. Its main producer is (line 3645):events.host(line 3698): forwardssession/created,agent/status,domain/changed, andAPI_REMOTE_FORWARDED_EVENTS.session/eventis emitted by the event-sourced session append innode_modules/@deepseek-ai/dsh-session/lib/index.jsappend()(line 1440-1480), which pushes to the session log and then emits"session/event"(line 1472). So the loop is:FrameQueue.iterate()has no backpressure, high-watermark, frame-rate limit, or frame dedup. It relies entirely on the assumption that the buffer will eventually drain. When a producer floods (suspected: session restore/resume re-emittingsession/event, or a re-forwarding feedback loop), that assumption breaks and the queue degenerates into a busy loop.Suggested fix
FrameQueue.iterate: after draining the buffer, if it was refilled within the same tick, yield to the event loop (e.g.await new Promise(setImmediate)/queueMicrotask) before continuing, so the loop cannot starve the process.buffer; when exceeded, coalescesession/eventframes persessionId(or drop oldest). This bounds the WebSocket traffic.(sessionId, event.seq): never forward the same session-event seq twice — this directly breaks any replay/feedback loop.Unverified (to be confirmed)
session/eventwas not fully isolated. The restore/resume path (dsh-session-persistenceresume/attachPrepared) and the synthetic-closer repair logic (dsh-session/lib/types/repair.js) are candidates but were not proven to loop.A focused repro around "resume a large session while the browser is connected" would confirm which producer is responsible.
All reactions