[Bug] dsh-llm-pi-ai ignores Retry-After when retrying — TPM-limited retries always fail

[netweaver]

[Bug] dsh-llm-pi-ai ignores Retry-After when retrying — TPM-limited retries always fail

Summary

When an OpenAI-compatible gateway accessed through dsh-llm-pi-ai (e.g. Volcano Ark) returns 429 / TPM rate limit, the retry uses the exponential backoff hard-coded in dsh-llm-pi-ai (default 500ms → 1000ms) and completely ignores the Retry-After header returned by the server.

Consequence: while the TPM limit is in effect, the retry interval is far shorter than the time needed for the quota to recover, so every retry necessarily hits 429 again; once the maxRetries budget is exhausted the request stops until a human triggers it again. This is not "missing retries" — it is "retrying at the wrong time and wasting the retry budget."

Impact

• Any user routing an OpenAI-compatible gateway (especially one that rate-limits by TPM, such as Volcano Ark) through dsh-llm-pi-ai.
• Compared with the official dsh-llm-deepseek adapter: the latter correctly reads retry-after and fills in providerRetryAfterMs, while dsh-llm-pi-ai does not read it.

Environment

• @deepseek-ai/dsh-llm-pi-ai: 0.1.0-rc.6 (also present in rc.7, see below)
• @deepseek-ai/dsh-llm-retry: 0.1.0-rc.6
• @earendil-works/pi-ai: 0.82.1
• Provider: Volcano Ark https://ark.cn-beijing.volces.com/api/v3, model deepseek-v4-flash-ga-260731, TPM limit 1000K/min
• Platform: Windows, DeepSeek Harness web profile

Steps to reproduce

1. Configure an OpenAI-compatible route (llm-pi-ai) in settings.yaml, with protocol openai-completions or openai-responses, and the model declaring reasoningEfforts (reasoning enabled).
2. Send several requests within a short window to trigger a server-side 429 (Volcano Ark returns ModelAccountTpmRateLimitExceeded).
3. Observe the retry behavior.

Actual observation (session-log llm/retry events)

{
  "retryId": "ab5c6832-c18b-468e-acee-8c8c75516b9f",
  "provider": "volcano-ark",
  "mode": "normal",
  "policyKey": "[\"normal\",2,[\"EMPTY_RESPONSE\",\"RATE_LIMIT\",\"SERVER\",\"TIMEOUT\",\"TRANSPORT\"],500,10000,0.1]",
  "retry": 1,
  "maxRetries": 2,
  "delayMs": 493.78,
  "failure": {
    "message": "429: {\"code\":\"ModelAccountTpmRateLimitExceeded\",\"message\":\"TPM (Tokens Per Minute) limit ...\"}",
    "code": "RATE_LIMIT"
  }
}

• delayMs is ~493ms, ~933ms, ~1072ms in sequence — i.e. exponential backoff from initialDelayMs=500 (500*2^0, 500*2^1).
• The failure object has no providerRetryAfterMs field.
• After 2 retries (retry=2/maxRetries=2) the budget is exhausted and turn/end ends with a RATE_LIMIT error; sending another message hits 429 again, in a loop.

Root-cause analysis

Error propagation chain:

Volcano Ark 429 + Retry-After header
   │
   ▼
pi-ai (openai-completions / openai-responses) stream
   │  catch → converted to an `error` event keeping only the string errorMessage,
   │  the original error object (including headers) is discarded
   ▼
dsh-llm-pi-ai toStreamChunks / mapStopReason
   │  → { kind:"error", failure:{ code:"RATE_LIMIT", message:"..." } }
   │  (no providerRetryAfterMs)
   ▼
outer dsh-llm-retry
   │  providerRetryAfterMs === undefined
   │  → uses localDelay (exponential backoff 500ms→1000ms)
   ▼
retry interval too short, TPM not recovered, 429 again

Key code points:

1. dsh-llm-pi-ai's profileOptions hard-codes maxRetries: 0, disabling pi-ai's internal retry (packages/llm/llm-pi-ai). Yet pi-ai's internal retry (getRetryDelayMs in provider-retry.js) already honors retry-after / retry-after-ms — this is the trade-off of handing retry responsibility over to the outer dsh-llm-retry.

2. Headers are lost during error propagation: pi-ai's stream (e.g. openai-responses.js) does output.errorMessage = formatOpenAIResponsesError(error) in the catch, turning the original error (with headers) into a string; AssistantMessage only has errorMessage?: string, no headers. As a result dsh-llm-pi-ai cannot obtain Retry-After, and the outer dsh-llm-retry's failure.providerRetryAfterMs is always undefined.

3. Comparison with the official adapter: dsh-llm-deepseek correctly reads and fills providerRetryAfterMs(response.headers.get("retry-after")) (see packages/llm/deepseek). So for the same gateway and the same 429, the official adapter honors Retry-After while the pi-ai adapter does not.

Expected behavior

On a 429, dsh-llm-pi-ai should propagate the server's Retry-After (or retry-after-ms) into failure.providerRetryAfterMs, so the outer dsh-llm-retry waits per the server's suggested delay instead of using a fixed exponential backoff.

Proposed solutions

Option A: Enable pi-ai's internal retry (change DSH's own package only)

Leverage the fact that pi-ai already honors Retry-After by having dsh-llm-pi-ai's profileOptions map profile.retryPolicy.maxRetries to pi-ai's maxRetries, thereby enabling pi-ai's internal retry (getRetryDelayMs in provider-retry.js natively honors retry-after / retry-after-ms).

Pros

• Changes only DSH's own package (packages/llm/llm-pi-ai); no third-party dependency touched.
• Reuses pi-ai's mature, correct, battle-tested Retry-After handling instead of reimplementing it in DSH.
• Small change surface (a single mapping in profileOptions).

Cons / risks

• Bypasses the outer dsh-llm-retry's configurability and observability: llm/retry events and UI presentation such as "retried model request (2/2)" would disappear or change.
• If the outer dsh-llm-retry also has a profile.retryPolicy configured, this can cause double retries (pi-ai inner + dsh-llm-retry outer), requiring a designed responsibility boundary (e.g. inner only waits on Retry-After, outer owns the final budget).
• Contradicts DSH's existing architecture direction of "unified outer retry"; may need a larger refactor rather than a point change.

Option B: Propagate providerRetryAfterMs (keep the outer retry's observability)

Have pi-ai's error event / AssistantMessage carry providerRetryAfterMs (parsed in normalizeProviderError or the stream catch from error.headers.get("retry-after") / retry-after-ms), and have dsh-llm-pi-ai's mapStopReason fill it into failure.providerRetryAfterMs.

Pros

• Preserves all existing capabilities of the outer dsh-llm-retry: configurability, observability (llm/retry events, UI presentation), and cross-provider consistency.
• Clearer semantics: the outer layer receives the server-suggested delay and decides the wait via providerRetryAfterMs — the path DSH already supports (the official dsh-llm-deepseek does exactly this).
• Aligns with the official adapter's behavior; predictable.

Cons / risks

• Changes span two layers: the third-party package @earendil-works/pi-ai (error propagation / types) plus DSH's own dsh-llm-pi-ai (propagating into the finish failure).
• pi-ai is a third-party dependency; the change breaks on upgrade, requiring re-patching or an upstream contribution.
• Requires extending the AssistantMessage / error-event types, or finding another way for the DSH layer to reach the original error — more engineering effort.

Comparison

Dimension	Option A	Option B
Change scope	DSH's own package only	pi-ai (third-party) + DSH, two layers
Touches third-party?	No	Yes (needs re-patch on upgrade)
Keeps outer observability	No (bypasses dsh-llm-retry)	Yes
Aligns with official adapter	No	Yes
Double-retry risk	Yes, needs design	No
Engineering effort	Small	Medium

Recommendation: if DSH wants to keep the "unified outer retry + observability" architecture, Option B is more correct (consistent with the official adapter), at the cost of changing the third-party pi-ai and contributing upstream; if a minimal change is preferred and moving the retry responsibility into pi-ai is acceptable, Option A is faster. Either way, the fix should not stop at "exponential backoff" without honoring Retry-After.

Additional notes

• This analysis was generated with the assistance of DSH.
• It has been confirmed that 0.1.0-rc.7 does not change this logic (e.g. resolveModelCompat); the issue is still present in rc.7.

[Bug] dsh-llm-pi-ai 重试时不遵循服务端 Retry-After，导致 TPM 限流下重试必然失败

摘要

使用 dsh-llm-pi-ai 接入的 OpenAI 兼容网关（如火山方舟）在遇到 429 / TPM 限流 时，重试使用的是 dsh-llm-pi-ai 硬编码的指数退避（默认 500ms → 1000ms），完全没有遵循服务端返回的 Retry-After 头。

后果：在 TPM 限流持续期间，重试间隔远小于配额恢复所需时间，每次重试必然再次命中 429，maxRetries 预算迅速耗尽后请求停止，直到人工再次触发。这不是"没有重试"，而是"重试时机错误、白费重试预算"。

影响

• 任何通过 dsh-llm-pi-ai 使用 OpenAI 兼容网关（尤其是按 TPM 限流的网关，如火山方舟）的用户。
• 与官方 dsh-llm-deepseek 适配器对比：后者正确读取 retry-after 并填充 providerRetryAfterMs，而 dsh-llm-pi-ai 不读取。

环境

• @deepseek-ai/dsh-llm-pi-ai：0.1.0-rc.6（rc.7 同样存在，见下）
• @deepseek-ai/dsh-llm-retry：0.1.0-rc.6
• @earendil-works/pi-ai：0.82.1
• 接入方：火山方舟 https://ark.cn-beijing.volces.com/api/v3，模型 deepseek-v4-flash-ga-260731，TPM 上限 1000K/min
• 平台：Windows，DeepSeek Harness web profile

复现步骤

1. 在 settings.yaml 配置一个 OpenAI 兼容路由（llm-pi-ai），协议 openai-completions 或 openai-responses，模型声明 reasoningEfforts（开启推理）。
2. 短时间内连续请求，触发服务端 429（火山方舟报 ModelAccountTpmRateLimitExceeded）。
3. 观察重试行为。

实际观察（会话日志 llm/retry 事件）

{
  "retryId": "ab5c6832-c18b-468e-acee-8c8c75516b9f",
  "provider": "volcano-ark",
  "mode": "normal",
  "policyKey": "[\"normal\",2,[\"EMPTY_RESPONSE\",\"RATE_LIMIT\",\"SERVER\",\"TIMEOUT\",\"TRANSPORT\"],500,10000,0.1]",
  "retry": 1,
  "maxRetries": 2,
  "delayMs": 493.78,
  "failure": {
    "message": "429: {\"code\":\"ModelAccountTpmRateLimitExceeded\",\"message\":\"TPM (Tokens Per Minute) limit ...\"}",
    "code": "RATE_LIMIT"
  }
}

• delayMs 依次为 ~493ms、~933ms、~1072ms —— 即 initialDelayMs=500 的指数退避（500*2^0、500*2^1）。
• failure 中没有 providerRetryAfterMs 字段。
• 重试 2 次后（retry=2/maxRetries=2）耗尽，turn/end 以 RATE_LIMIT 错误结束；人工再发消息又命中 429，循环往复。

根因分析

错误传递链：

火山方舟 429 + Retry-After 头
   │
   ▼
pi-ai (openai-completions / openai-responses) 的 stream
   │  catch → 转成 error 事件，只保留字符串 errorMessage，
   │  丢弃原始错误对象（含 headers）
   ▼
dsh-llm-pi-ai 的 toStreamChunks / mapStopReason
   │  → { kind:"error", failure:{ code:"RATE_LIMIT", message:"..." } }
   │  （无 providerRetryAfterMs）
   ▼
外层 dsh-llm-retry
   │  providerRetryAfterMs === undefined
   │  → 使用 localDelay（指数退避 500ms→1000ms）
   ▼
重试间隔太短，TPM 未恢复，再次 429

关键代码点：

1. dsh-llm-pi-ai 的 profileOptions 硬编码 maxRetries: 0，禁用了 pi-ai 的内部重试（packages/llm/llm-pi-ai）。而 pi-ai 的内部重试（provider-retry.js 的 getRetryDelayMs）本来就会遵循 retry-after / retry-after-ms——这是设计上把重试统一交给外层 dsh-llm-retry 的取舍。

2. 错误透传丢失 headers：pi-ai 的 stream（如 openai-responses.js）在 catch 里 output.errorMessage = formatOpenAIResponsesError(error)，把原始错误（含 headers）转成字符串；AssistantMessage 只有 errorMessage?: string，没有 headers。因此 dsh-llm-pi-ai 拿不到 Retry-After，外层 dsh-llm-retry 的 failure.providerRetryAfterMs 恒为 undefined。

3. 对比官方适配器：dsh-llm-deepseek 的 providerRetryAfterMs(response.headers.get("retry-after"))（见 packages/llm/deepseek）正确读取并填充。所以同样的网关、同样的 429，官方适配器遵循 Retry-After，pi-ai 适配器不遵循。

期望行为

dsh-llm-pi-ai 在遇到 429 时，应把服务端 Retry-After（或 retry-after-ms）透传到 failure.providerRetryAfterMs，让外层 dsh-llm-retry 按服务端建议的延迟等待，而不是用固定指数退避。

建议方案

方案 A：启用 pi-ai 内部重试（改动 DSH 自己的包）

利用 pi-ai 已遵循 Retry-After 的能力，让 dsh-llm-pi-ai 的 profileOptions 把 profile.retryPolicy.maxRetries 映射到 pi-ai 的 maxRetries，从而启用 pi-ai 内部重试（provider-retry.js 的 getRetryDelayMs 天然遵循 retry-after / retry-after-ms）。

优点

• 只改 DSH 自己的包（packages/llm/llm-pi-ai），不碰第三方依赖。
• 直接复用 pi-ai 成熟、正确、经过考验的 Retry-After 处理逻辑，无需在 DSH 侧重复实现。
• 改动面小（profileOptions 一处映射）。

缺点 / 风险

• 绕过外层 dsh-llm-retry 的可配置与可观测性：llm/retry 事件、UI 上「已重试模型请求 (2/2)」这类呈现会消失或改变。
• 若外层 dsh-llm-retry 的 profile.retryPolicy 也配置了重试，可能产生双重重试（pi-ai 内层 + dsh-llm-retry 外层），需要设计好职责边界（例如：内层只在遵循 Retry-After 时等待，外层负责最终预算）。
• 与 DSH「统一由外层重试」的既有架构方向相悖，可能需要较大重构而非单点改动。

方案 B：透传 providerRetryAfterMs（保留外层重试的可观测性）

让 pi-ai 的 error 事件 / AssistantMessage 携带 providerRetryAfterMs（在 normalizeProviderError 或 stream 的 catch 处从 error.headers.get("retry-after") / retry-after-ms 解析），由 dsh-llm-pi-ai 的 mapStopReason 填充到 failure.providerRetryAfterMs。

优点

• 保留外层 dsh-llm-retry 的全部现有能力：可配置、可观测（llm/retry 事件、UI 呈现）、跨 provider 一致。
• 语义更清晰：外层拿到服务端建议的延迟，用 providerRetryAfterMs 决定等待，是 DSH 已有的、受支持的路径（官方 dsh-llm-deepseek 正是这么做的）。
• 与官方适配器行为对齐，行为可预期。

缺点 / 风险

• 改动跨两层：需要改第三方包 @earendil-works/pi-ai（error 透传 / 类型）+ DSH 自己的 dsh-llm-pi-ai（透传到 finish failure）。
• pi-ai 是第三方依赖，改动升级即失效，需重新打补丁或向 pi-ai 上游提交。
• 需要扩展 AssistantMessage / error 事件类型，或换一种在 DSH 层能拿到原始错误的方式，工程上更繁琐。

对比结论

维度	方案 A	方案 B
改动范围	仅 DSH 自己的包	pi-ai（第三方）+ DSH 两层
是否需改第三方	否	是（升级需重打）
保留外层可观测性	否（绕开 dsh-llm-retry）	是
与官方适配器对齐	否	是
双重重试风险	有，需设计	无
工程量	小	中

倾向建议：若 DSH 希望维持「外层统一重试 + 可观测」的架构，方案 B 更正确（与官方适配器一致），代价是要改第三方 pi-ai 并向上游提交；若希望最小改动、接受重试职责前移到 pi-ai，则方案 A 更快。两者都不应只停在「指数退避」而不遵循 Retry-After。

补充说明

• 以上内容是dsh帮忙分析生成
• 已确认 0.1.0-rc.7 中 resolveModelCompat 等未改动此逻辑，问题在 rc.7 仍存在。

[netweaver]

llm-pi-ai 插件的讨论是应该发在这里吗？ 没找到其他地方可以讨论

[weijiafu14]

Measurement that narrows this one layer down, in case it helps the fix land in the right place.

We fault-injected the real pi-ai 0.84.1 transport (crafted wire responses into the unmodified client + retryProviderRequest; nothing else synthetic): a 429 with Retry-After: 2 is honored — the retry waited ~2.0s and recovered on the next attempt. Server-requested delays above the 60s default cap fail loudly naming the delay, and the cap is caller-raisable (maxRetryDelayMs) — a minute-scale window under a raised cap also recovered in the same run. Harness + verdicts: retry-experiment.mjs / retry-experiment-results.json.

So pi-ai's own retry machinery already does what your report asks for. The exponential-backoff-only behavior you measured on the dsh-llm-pi-ai route means the wrapper's retry layer is superseding (or re-driving) the transport's — which matches your diagnosis, and suggests the smallest fix is in that wrapper: either let pi-ai's transport-level retry run with maxRetries/maxRetryDelayMs passed through, or teach the wrapper's classifier to read Retry-After before applying its own schedule.

Scope note: our runs are against the transport seam, not your Volcano Ark gateway specifically — but the TPM shape (429 + Retry-After) is exactly what the injection reproduces.

[MajidAsghariTabrizi]

We hit a related issue from the provider side (free-model tier) and ended up building free-best-router — a small OpenAI-compat routing layer that surfaces upstream Retry-After and the soonest-to-recover model in its 502/429 response.

What we emit on a saturated pool (server.mjs):

• Retry-After: <seconds> — computed from the soonest-to-expire cooldown, clamped to [1, 600] s
• x-free-router-next-eligible: <provider>/<model> — the model that will be eligible first
• 502 body with next_eligible_at (ms epoch) and next_eligible_model
• 429 body (when the failure type is rate_limit) with retry_after_seconds and next_eligible_model

If llm-pi-ai could read either of these signals upstream of its own retry loop, blind 429 retry storms become bounded. Happy to send a small PR to wire it.

Concrete suggestion: when the openai-completions adapter gets a 429/503 with a Retry-After header, it should:

1. Honour the header (current bug)
2. Pass it to the caller via the response headers so downstream routers can chain
3. Optionally short-circuit the retry and surface the next_eligible_model hint to the agent

Full write-up in our DSH Show Your Plugins! discussion #5508.