[Bug] dsh-llm-pi-ai: Codex outages classified PI_AI_ERROR, never retried

[verdier]

Version: @deepseek-ai/dsh 0.2.0-rc.2 (same code on main, packages/llm/llm-pi-ai/src/stream.ts, classifyPiAiError).

What happens: on the openai-codex route, Codex reports server-side outages as in-stream error events, with no HTTP status in the text. classifyPiAiError falls through to PI_AI_ERROR, which is not in the default retryableCodes, so the turn ends instead of being retried. Messages seen in our session logs over four days (24 failed turns, all at step start with zero output tokens, several sessions failing in the same minute):

• Codex error: An error occurred while processing your request. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (16)
• Codex error: Our servers are currently overloaded. Please try again later. (4)
• WebSocket closed 1000 (3)
• WebSocket error (1)

The /\b5\d\d\b/ test never matches these, and \bsocket\b does not match WebSocket (no word boundary before socket), while the existing WebSocket closed unexpectedly pattern does not cover WebSocket closed 1000.

Suggested fix (in classifyPiAiError, before the final fallback):

if (/servers are currently overloaded|an error occurred while processing your request/i.test(message)) return 'SERVER'
if (/\bWebSocket (?:error|closed)\b/i.test(message)) return 'TRANSPORT'

Workaround: per-route retryPolicy.retryableCodes including PI_AI_ERROR on openai-codex. It works, but it also retries non-transient PI_AI_ERRORs (content-policy refusals, pending/deferred).

[verdier]

Follow-up on the root cause, and a simpler fix than the two regexes above.

Why text matching at all: these failures arrive mid-stream (WebSocket open or SSE already 200), so there is no HTTP status or Retry-After to read. The Codex event does carry a structured error.code: pi-ai reads it into CodexApiError.code/.payload (extractCodexEventError), then flattens everything to AssistantMessage.errorMessage (openai-codex-responses.ts, output.errorMessage = formatProviderError(normalizeProviderError(error))). The code never reaches dsh-llm-pi-ai. pi's maintainers keep text classification as the contract (earendil-works/pi#7444, closed: "the coding-agent does retry in the outer loop").

Suggested fix: pi-ai already exports that outer-loop classifier, isRetryableAssistantError(message: AssistantMessage) (@earendil-works/pi-ai, utils/retry.ts, present in 1.1.0). Its pattern list covers all four messages reported here (you can retry your request, overloaded, websocket.?closed, websocket.?error) and is maintained with each provider's wording. mapStopReason could consult it before falling back to PI_AI_ERROR, so DSH stops keeping a second, drifting copy of the list:

case 'error': {
  const text = message.errorMessage ?? 'pi-ai stream error'
  const code = classifyPiAiError(text)
  return { kind: 'error', failure: { message: text, code: code === 'PI_AI_ERROR' && isRetryableAssistantError(message) ? 'SERVER' : code } }
}

[PerryLink]

你的诊断逐条成立，而且你那条 follow-up 已经找到了比自建正则更好的路。我把链路上每一环的行号补齐，并给你一个决定性的版本事实：你建议的那个导出，在仓库现有的依赖范围内今天就能用，不需要改 manifest。

1. 分类函数：你的三处断言逐行核实

packages/llm/llm-pi-ai/src/stream.ts 的 classifyPiAiError（:42-68）：

• :50 → if (/\b5\d\d\b/.test(message)) return 'SERVER'
  ⇒ 数字-only 的 5xx 分支 —— 你写的"/\b5\d\d\b/ 永不匹配这些消息"成立（Codex 的文案里没有状态码）。
• :59-60 → if (/\b(?:network|connection|socket|fetch)\b|\bECONN[A-Z]+\b/i.test(message) || /\b(?:other side closed|HTTP2 request did not get a response|WebSocket closed unexpectedly)\b/i.test(message)
  ⇒ 你指出的两个缺口都对：\b 在 WebSocket 里的 Web 与 Socket 之间没有词边界（两侧都是词字符），所以 \bsocket\b 不匹配；而 WebSocket closed 1000 也不等于 WebSocket closed unexpectedly。
• :67 → return 'PI_AI_ERROR' ⇒ 兜底，你的判断正确。

同函数里已有的近亲分支供你对照：:43 AUTH、:44 QUOTA_EXCEEDED_CODE、:45 RATE_LIMIT、:48-49 INVALID_REQUEST、:51 TIMEOUT、:58 stream ended (?:before|without) → TRANSPORT、:64 terminated|premature close → TRANSPORT。

2. 重试闸门：PI_AI_ERROR 确实不在默认集合里

• packages/llm/llm/src/retry-policy.ts:14 → const DEFAULT_MAX_RETRIES = 5
• :18-24 →

const DEFAULT_RETRYABLE_CODES = Object.freeze([
  EMPTY_RESPONSE_CODE,
  'RATE_LIMIT',
  'SERVER',
  'TIMEOUT',
  'TRANSPORT',
])

⇒ PI_AI_ERROR 不在其中，你的判断成立。

• 闸门在 packages/llm/llm-retry/src/index.ts:215 → } else if (!policy.retryableCodes.includes(failure.code)) {
• ⚠️ 一处路径更正：retry-policy.ts 在 packages/llm/llm/src/，不是 packages/llm/llm-retry/src/。你若在报告里引路径，请用前者。
• 你的 workaround 也核实了：packages/llm/llm/src/retry-policy.ts:90 → retryableCodes: z.array(z.string()).default([...DEFAULT_RETRYABLE_CODES]) ⇒ 确实可按 route 覆盖，与你说的行为一致（包括"会连非瞬时的 PI_AI_ERROR 一起重试"这个副作用）。

3. ⭐ 决定性事实：你建议的导出在现有依赖范围内就能用

你写 isRetryableAssistantError "present in 1.1.0"。我核了两件事：

(a) 仓库声明的范围是 ^1.0.2 —— packages/llm/llm-pi-ai/package.json → "@earendil-works/pi-ai": "^1.0.2"。

(b) 该范围内的最新版是 1.2.0 —— registry 上 1.0.0 / 1.0.1 / 1.0.2 / 1.0.3 / 1.0.4 / 1.1.0 / 1.2.0，latest = 1.2.0。

(c) isRetryableAssistantError 确实被导出 —— @earendil-works/pi-ai@1.2.0 的 dist/utils/retry.d.ts:57：

export declare function isRetryableAssistantError(message: AssistantMessage): boolean;

（同文件 :47 还有 retryAssistantCall(...)，注释明确说它是 policy-driven 的外层重试循环，:37 提到"a non-retryable error (per {@link isRetryableAssistantError}, including quota/…)"。）

⇒ 结论：^1.0.2 这个 caret 范围今天解析到 1.2.0，因此你那条 mapStopReason 改法不需要动 manifest，可以直接在现有依赖下实现。

⚠️ 但请在你的报告里写明一处前提：能否 import 取决于实际解析到的版本。若某个 lockfile 把 pi-ai 钉在 1.0.2，则该导出可能不存在（它自 1.1.0 起才有）。所以建议在诉求里加一句"需要 pi-ai >= 1.1.0，当前 ^1.0.2 范围已满足，但请确认解析版本"。

4. 你为什么应该优先推 isRetryableAssistantError 而不是自己加正则

你的 follow-up 已经给出了理由（pi 维护者把文本分类当契约、regex 列表随 provider 措辞维护），我同意，并补一条来自本仓的证据：这个函数里已经有一份"跟着上游措辞漂移"的列表 —— :54-56 的注释逐一列出了四种不同 provider 的截断措辞（… stream ended before message_stop、… before a terminal response event、… ended without a terminal event、Stream ended without finish_reason），:59-64 又收了 undici 的 terminated 与 Node 的 Premature close。

⇒ 这一处正是"第二个会漂移的副本"的现场证据，用它支持你的主张会很有力：每加一个 provider/上游改一次措辞，这里就要再补一条正则。

5. 一处边界（按规矩标注）

• 我核的是 origin/master = d743267388（= dsh-v0.2.1-alpha.2）上的 stream.ts:42-68、llm/src/retry-policy.ts:14/18-24/90、llm-retry/src/index.ts:215，以及 registry 上 @earendil-works/pi-ai 的版本与 1.2.0 的 dist/utils/retry.d.ts。
• ⚠️ 我没有核实你日志里那 24 个失败轮次的具体条目（那需要你的 session log），也没有核实 pi-ai 侧 extractCodexEventError / openai-codex-responses.ts 的内部路径 —— 那是另一个包，不在本仓。第 2 节 follow-up 里关于 error.code 在 flatten 时丢失的描述，我按你的陈述采信，未独立验证。
• 1.2.0 的 retry.d.ts 我是从 CDN 取的发布产物，不是从本仓的 lockfile 推导 —— 所以你本机的解析版本请自己确认。

6. 建议的诉求写法（把三层分开）

1. 缺陷：openai-codex 路线的服务端过载/错误经 classifyPiAiError 落到 PI_AI_ERROR，而它不在 DEFAULT_RETRYABLE_CODES ⇒ 整轮直接结束（你的 24 个失败轮、零输出 token 是证据）。
2. 修法：mapStopReason 的 'error' 分支在落到 PI_AI_ERROR 前先问 isRetryableAssistantError(message)（pi-ai 导出，>= 1.1.0；当前 ^1.0.2 已能满足）—— 让 DSH 不再维护第二份会漂移的列表。
3. workaround（临时）：按 route 覆盖 retryPolicy.retryableCodes 加入 PI_AI_ERROR，并注明副作用（会重试非瞬时的 PI_AI_ERROR，如内容策略拒绝与 pending/deferred）。

[verdier]

Thanks for checking it line by line.

Two corrections to the report:

• The default list lives in packages/llm/llm/src/retry-policy.ts (DEFAULT_RETRYABLE_CODES, :18-24), not in llm-retry. The gate that reads it is packages/llm/llm-retry/src/index.ts:215.
• isRetryableAssistantError was added in pi-ai 1.1.0. The current ^1.0.2 range already resolves to 1.2.0, so the mapStopReason change needs no manifest edit, but the lockfile has to resolve to at least 1.1.0. Bumping the floor to ^1.1.0 would make that explicit.

The request is still the one in the follow-up above: on 'error', ask isRetryableAssistantError(message) before falling back to PI_AI_ERROR, so DSH stops keeping its own copy of the provider wording list.