Repository navigation
Replies: 1 comment
|
在 DSH Desktop 0.2.2-rc.3 的内置 pi-ai 适配器上复现了同样的行为。用合成的 OpenAI Completions 模拟流,从 0 ms 起每 100 ms 发
billion-context 会在代理预压缩时发送同款 SSE 注释,因此这也会影响等待本地代理准备请求的情况。插件侧的折叠进展及重试测试已记录到 billion-context #3978。上述三个用例直接调用桌面发行版的适配器,只用了本机模拟上游。 |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Summary
dsh-llm-pi-ai's stream-idle watchdog measures time to the next harnessStreamChunk, not transport activity. A server that is still inside a long prompt pass (prompt processing / prefill) keeps the connection alive with SSE comment frames (: ping) and sends no data frames, so the watchdog sees total silence. With the defaultstreamIdleTimeoutMs = 300_000, any first token later than five minutes is reported as a dead stream and the request is aborted — the connection, the server's work and the retry budget are all discarded:The twin adapter in this repo does not have this failure mode:
dsh-llm-deepseekparses SSE itself withEventSourceParserStream({ onComment })and callswatchdog.pulse()on every comment frame.IdleWatchdog.pulse()is documented for exactly this case — "Rearm an outstanding demand after transport activity that yields no iterator value" (packages/util/timeout/src/index.ts) — butdsh-llm-pi-ainever calls it.Why the pi-ai path cannot see the keep-alives
Three layers drop the traffic that proves the stream is alive:
: pingfetchbodyTimeoutopenaiSDKSSEDecoderline.startsWith(':')adds no data; the closing blank line returnsnullwhen!event && !data.lengthopenai-completionsstartevent yields no contentdsh-llm-pi-aitoStreamChunkscase 'start': breakSo the first chunk the watchdog can see is the first content / reasoning / tool-call delta, which makes window #1 effectively time to first token, and the field name (
streamIdleTimeoutMs, "idle") misleading.Related: the retry layer amplifies the damage.
dsh-llm-retrydefaults tomaxRetries: 5withretryableCodesincludingTIMEOUTand a 500 ms → 10 s backoff, so a deterministictimeout < time-to-first-tokenturns into six full prompt passes queued on the same deployment, which further lengthens the queue that caused the timeout.Observed on a real deployment
vLLM-class server (1M-token context model, two DGX Spark nodes), raw wire capture of a 200k-token prompt pass:
A 2k-token request issued 4 s into that pass received headers in 12 ms and its first token after 88.5 s, again with
: pingframes every ~25 s: bytes flow continuously, data frames do not.Server counters after 815 finished requests:
The 300 s deadline was firing on requests the server was still working on, and each retry re-queued the same prompt.
Proposed fix (patch + regression test)
Wrap the platform
fetchfor the adapter's stream calls and pulse the watchdog on every received body byte. pi-ai already forwardsoptions.fetchinto the OpenAI client, so this needs no pi-ai change:The regression test lives in
packages/llm/llm-pi-ai/tests/adapter.spec.ts(mock-server.tsgains an opt-inpingEveryMswriter). The server sends the role chunk, then: pingevery 100 ms, and the first content frame only after 600 ms, withstreamIdleTimeoutMs: 400: without the fix the call ends inTIMEOUT, with it the call completes. Both directions verified by removing thefetch:option again.Local validation (Windows 11, Node 24,
pnpm install --filter "@deepseek-ai/dsh-llm-pi-ai..." --ignore-scripts):pnpm exec vitest run packages/llm/llm-pi-ai→ 20 files / 445 tests pass, including the new testpnpm exec tsc -b packages/llm/llm-pi-ai/tsconfig.json→ cleanpnpm exec tsx scripts/run-oxlint.ts packages/llm/llm-pi-ai→ cleanNotes / design alternatives
{ type: 'ping' }) and pulsing fromtoStreamChunkswould be more explicit; it is a pi-ai change, though — both0.85.1and1.4.0drop comment frames today. Thefetchtap keeps the fix inside this repo and covers every pi-ai API, not onlyopenai-completions(Anthropic Messages and Responses transports also send keep-alives).dsh-llm-deepseek.streamIdleTimeoutMsper provider and trim the retry loop for deterministic timeouts. That hides the bug rather than fixing it, and it leaves deployments that keep the connection alive with comments (vLLM: ping, LiteLLM, and most SSE gateways) permanently mis-classified.CONTRIBUTING.mdsays external pull requests are not accepted yet, so I am posting the diagnosis and the patch here. Happy to reshape it (split first-chunk budget, config docs, changeset) or provide a patch file if that is easier to review.中文摘要
报错串来自
dsh-llm-pi-ai自己的idleWatchdog,它只统计"下一个 harnessStreamChunk",不统计网络活动:SSE 的: ping注释帧被openaiSDK 的SSEDecoder丢弃,pi-ai 的start(响应头到达)事件又被toStreamChunks的case 'start': break吃掉,所以长时间的 prompt pass(prefill)看起来和断流一模一样,默认streamIdleTimeoutMs = 300000到点就 abort。dsh-llm-deepseek因为用EventSourceParserStream({ onComment }) → watchdog.pulse()所以没有这个问题,pulse()本身也正是为"有传输活动但没有迭代器值"设计的。实测:vLLM 侧 prefill 期间每 ~25s 发
: ping;200k tokens 首个 token 93.6s;大 prefill 期间一个 2k tokens 的请求要等 88.5s(只有心跳、没有 data 帧);服务端 815 次请求里有 3 次 TTFT 超过 300s、28 次客户端中途断开。再叠加dsh-llm-retry默认 5 次重试,会把同一个 prompt 反复塞回同一台机器,越重试队列越长。补丁:把传给 pi-ai 的
options.fetch包一层,收到任何 body 字节就watchdog.pulse()(pulse()本来就只对未决的next()生效,空闲时是 no-op)。新增单测覆盖:带: ping心跳的 600ms 首帧延迟在 400ms 空闲预算下应当成功,去掉补丁则TIMEOUT。包内 445 个测试 +tsc -b+oxlint全绿。All reactions