429 with insufficient_quota is classified as terminal QUOTA and never retried, breaking gateway/relay API users

[TheOnlyShuaiBoss]

Environment: latest checkout, running via node --import tsx/esm apps/cli/src/bin.ts web

Problem:
A 429 response whose body contains code: "insufficient_quota" / "Allocated quota exceeded" is mapped to the terminal code QUOTA by httpErrorCode() in packages/llm/llm-deepseek/src/adapter.ts (the isQuotaExceededError check runs BEFORE the status === 429 branch). Because QUOTA is not in DEFAULT_RETRYABLE_CODES in packages/llm/llm/src/retry-policy.ts, llm-retry skips retrying entirely — a single transient 429 kills the whole turn.

Why it matters:
Many OpenAI-compatible gateways/relays (one-api, new-api, chatanywhere, etc.) return 429 insufficient_quota for TRANSIENT rate/allowance throttling, not just for truly exhausted balances. For those users every burst of requests fails with no retry, making the harness unusable under rate limits.

Repro:

• Configure a deepseek/openai-compatible provider whose gateway returns 429 + insufficient_quota on burst
• Observe session event: failure.code === 'QUOTA', no llm/retry event, turn errors out
• Expected: 429 should follow the RATE_LIMIT retry path (respecting Retry-After), and only genuinely exhausted-balance responses should be terminal

Suggested fix:

1. Add 'QUOTA' to DEFAULT_RETRYABLE_CODES (worked for us)
2. Or better: in httpErrorCode, only classify as QUOTA when it is a 4xx/credit-style exhaustion (e.g. 400/403 semantics), and let status === 429 take precedence for RATE_LIMIT

[Yo1ick]

Confirmed the mechanism you describe, with one correction to the proposed fix and a primary-source payload that I think changes the shape of the right solution.

Your root cause is right

httpErrorCode (packages/llm/llm-deepseek/src/adapter.ts) evaluates the text classifier before it looks at the status:

if (isQuotaExceededError(detail)) return QUOTA_EXCEEDED_CODE
if (status === 429) return 'RATE_LIMIT'

classifyPiAiError (packages/llm/llm-pi-ai/src/stream.ts) has the same order. So a text match outranks a 429, and QUOTA is not in DEFAULT_RETRYABLE_CODES (EMPTY_RESPONSE, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT). Relay users get a terminal turn where the upstream meant "come back in a moment".

Your deployment can be fixed today, without a default change

retryPolicy is provider config, and retryableCodes is part of it, so this is a patch-layer change on your side — no upstream change required:

# $DSH_HOME/profiles/<name>/cordis.patch.yml
- id: llm-deepseek
  config:
    # an id-targeted patch replaces the entry's whole config — restate your other fields
    retryPolicy:
      mode: normal   # required by the schema
      retryableCodes: [EMPTY_RESPONSE, RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT, QUOTA]

Verified against RetryPolicySchema on rc.7: it resolves to maxRetries: 2, backoff 500ms → 10s, jitterRatio 0.1. Omitting mode is rejected by the union, which is the one thing easy to get wrong here.

Why QUOTA should not become retryable by default

QUOTA is a classification, not a policy. Other surfaces consume it to tell the user to top up; making it retryable moves every genuinely terminal balance failure to 3 attempts and weakens that signal for everyone in order to fix one deployment shape. The asymmetry also runs the wrong way: an unnecessary retry costs a little latency, while a terminal failure that should have been retried costs the request. Precision belongs in the classifier, not in the retry set.

Why "400/403 means real exhaustion" cannot work either

Terminal exhaustion has no consistent status across providers. From my own instance's session log (rc.7, deepseek-official, deepseek-v4-flash):

{"type":"turn/end","data":{"turn":1,"reason":{"kind":"error",
  "error":{"message":"Insufficient Balance","code":"QUOTA","status":402}}}}

DeepSeek's own hard stop is 402. Alongside the two other majors:

provider	terminal exhaustion wording	status
DeepSeek	Insufficient Balance	402
OpenAI	insufficient_quota / You exceeded your current quota…	429
Anthropic	Your credit balance is too low to access the API…	400

402, 429 and 400 all mean "terminal" somewhere, and 429 means both things depending on the body. Two consequences:

1. httpErrorCode has no 402 branch at all (grep -rn "\b402\b" packages/llm → no match). The record above is classified correctly only because the message text happens to match /\binsufficient[\s_-]+(?:quota|balance|credits?)\b/; the 402 contributed nothing. A wording change upstream turns it into HTTP_402.
2. Anthropic's wording is missed today — credit balance is too low has the noun but no exhaustion verb, so none of the five patterns fire.

The classifier also errs in the opposite direction

Rate limits are commonly expressed as quota. Google-style APIs and OpenAI-compatible relays emit wording like:

Quota exceeded for quota metric 'Generate Content API requests per minute'

This matches /\b(?:quota|usage[\s_-]+limit)[\s_-]+(?:exceeded|exhausted|reached)\b/ → QUOTA → terminal, never retried. Same bug as yours, opposite direction, and it survives whatever the retry set does. The root problem is that quota denotes both an account balance and a rate allowance, and the five patterns treat them as one thing.

Multi-clause detail makes it worse. detail is [code, type, message].join(' '), so a single string can carry both readings — Rate limit exceeded. Your quota resets in one minute. — and any proximity-based rule that ignores clause boundaries will call that terminal.

A proximity-only fallback is not enough

The obvious repair for the missed wording is a fallback that pairs a quota noun with an exhaustion marker inside a token window. I tried exactly that on rc.7 (window 8, tokens split on [\s,_-]+, whole detail string) and measured it against a 19-case matrix. It fixes insufficient account balance and keeps all seven existing assertions in service.spec.ts green, but three cases fail:

detail	expected	proximity-only
Your credit balance is too low to access the API	terminal	missed (noun present, no exhaustion verb)
Rate limit exceeded. Your quota resets in one minute.	transient	terminal — exceeded and quota are 2 tokens apart across a sentence boundary
Quota exceeded for quota metric '… requests per minute'	transient	terminal

The second row is a new failure the fallback introduces, and it is the expensive direction. It survives a negative-case suite made of single-clause strings (rate limit reached, quota resets in one minute, context window exceeded) because each of those lacks one of the two token classes — such negatives verify the rule's definition rather than its risk surface. Clause scoping and a transient veto are what actually contain it.

Proposed shape

Structured signal first, prose last, and bias unknown toward retryable:

• T0 — exact code/type enums (insufficient_quota, insufficient_balance, credit_balance_too_low, billing_hard_limit_reached vs rate_limit_exceeded, overloaded_error). These are provider-maintained values and deserve exact matching; joining them into one string with free prose discards that precision.
• T1 — unambiguous statuses: 402 → QUOTA.
• T2 — prose, split into wording that has no transient reading (insufficient balance, balance exhausted/depleted, out of credits, balance is too low, run out of credits) and wording that does (quota exceeded/reached), where the ambiguous half is vetoed by transient markers in the same clause (retry, try again, resets, per minute/hour/day, RPM/TPM, too many requests, slow down, overloaded). Clause-scoped, window 4 tokens. Deliberately no status numbers in the veto list, since OpenAI ships terminal insufficient_quota on 429.
• T3 — fall through to the status default (429 → RATE_LIMIT).

Prose layer, same signature as today (drop-in for isQuotaExceededError)

const HARD_TERMINAL_RE = [
  /\binsufficient[\s_-]+(?:quota|balance|credits?|funds?)\b/i,
  /\b(?:balance|credits?)[\s_-]+(?:exhausted|depleted|drained)\b/i,
  /\bout[\s_-]+of[\s_-]+(?:credits?|budget)\b/i,
  /\b(?:balance|credits?|funds?)\b[^.;]{0,24}\bis[\s_-]+too[\s_-]+low\b/i,
  /\brun(?:s|ning)?[\s_-]+out[\s_-]+of[\s_-]+(?:credits?|quota|balance|funds?)\b/i,
  /\bno[\s_-]+(?:remaining|available)[\s_-]+(?:credits?|quota|balance|funds?)\b/i,
]
const SOFT_TERMINAL_RE = [
  /\b(?:quota|usage[\s_-]+limit)[\s_-]+(?:exceeded|exhausted|reached)\b/i,
  /\bexceed(?:ed|s)?[\s_-]+(?:(?:your|the)[\s_-]+)?(?:current[\s_-]+)?quota\b/i,
]
const TRANSIENT_VETO_RE =
  /\b(?:retry|try[\s_-]+again|resets?|slow[\s_-]+down|too[\s_-]+many[\s_-]+requests|overloaded|per[\s_-]+(?:second|minute|hour|day)|[rt]pm|requests?[\s_-]+per)\b/i

export function isQuotaExceededError(detail: string): boolean {
  return detail.split(/[.;\n!]/).some((clause) => {
    if (clause.trim() === '') return false
    if (HARD_TERMINAL_RE.some(re => re.test(clause))) return true   // never vetoed
    if (TRANSIENT_VETO_RE.test(clause)) return false
    return SOFT_TERMINAL_RE.some(re => re.test(clause)) || coOccursInClause(clause)
  })
}

coOccursInClause pairs a quota noun (quota|balance|credits?|funds?|allowance|budget) with an exhaustion marker (insufficient|exhausted|depleted|drained|exceed(ed|s)?) within 4 tokens of the same clause, which is what catches wording like insufficient account balance that the fixed patterns miss.

Checked against 19 cases: the 7 assertions currently in packages/llm/llm/tests/service.spec.ts are unchanged, plus DeepSeek 402, Anthropic too low, insufficient account balance, Insufficient balance, please top up and try again (hard wording must beat the veto), and the transient set above including Rate limit exceeded. Your quota resets in one minute. and the requests per minute quota wording.

Related reports

• Bug: pi-ai adapter misclassifies 401/403 in error text as AUTH, surfacing misleading "API key is invalid" #3073 — pi-ai adapter misclassifies 401/403 appearing in error text as AUTH
• 【BUG反馈】额度耗尽（403）被统一显示成 "API key is invalid" #1631 — quota exhaustion (403) surfaced with the wrong message
• Fix: classify 401/403 bodies by semantics (context overflow / quota) instead of always AUTH #1127 — classify 401/403 bodies by semantics

These four are the same root pattern: provider error semantics inferred from prose. classifyPiAiError says so in its own comment — "we are left pattern-matching terse words here" — because pi-ai flattens the error to message upstream. That path genuinely has only text, so the prose layer has to be precise on its own; the deepseek path, by contrast, already receives code, type and status and currently throws the distinction away by joining them.

Happy to break the T0/T1 part out into its own thread if that is easier to act on. Aware that external PRs are not being accepted right now, so this is a proposal rather than a patch.

中文摘要

你定位的顺序问题是对的：httpErrorCode 里文本分类跑在 status === 429 之前，classifyPiAiError 同样如此，而 QUOTA 不在 DEFAULT_RETRYABLE_CODES 里。

但两个建议我认为方向不对：

1. 你的场景不需要改默认值。 retryPolicy.retryableCodes 本来就是 provider 配置，在自己的 cordis.patch.yml 里把 QUOTA 加进去即可（mode: normal 是 schema 必填，漏了会被 union 拒绝；我在 rc.7 上验证过）。把 QUOTA 加进全局默认，等于让所有真·余额耗尽都变成 3 次尝试，并削弱其它消费方依赖的终态语义——QUOTA 是分类，不是策略。
2. 「400/403 表示真耗尽」不成立。 各家状态码不一致：DeepSeek 是 402（我本机 session 日志实录 {"code":"QUOTA","status":402}），OpenAI 的 insufficient_quota 走 429，Anthropic 的 credit balance is too low 走 400。

顺带两个现存缺陷：httpErrorCode 完全没有 402 分支（上面那条判对纯粹是靠正则匹到了 insufficient balance 字符串）；而 Quota exceeded for quota metric '... requests per minute' 这类速率限制会被判成终态，方向和你的问题正好相反。

建议的形状是分层：code/type 枚举精确匹配 → 无歧义状态码（402）→ 散文（拆成"无瞬时读法"和"有歧义"两类词表，后者受同子句瞬时标记否决）→ 兜底偏向可重试。

[TheOnlyShuaiBoss]

Subject: Re: 429/insufficient_quota misclassification on relay gateways — confirmed + one residual

Hello,

Thank you for the detailed root-cause analysis. It is accurate and we have since implemented the fixes you outlined, with one residual issue for your awareness.

What we confirmed (matching your diagnosis):

httpErrorCode (llm-deepseek) and classifyPiAiError (llm-pi-ai) both ran the prose quota classifier before consulting the HTTP status, so a text match outranked a 429, and QUOTA was not in DEFAULT_RETRYABLE_CODES.
On our relay (token.sensenova.cn, OpenAI-compatible), 429 insufficient_quota with body "Allocated quota exceeded / inference tpm exhausted" was being treated as terminal and never retried.
What we did (deployment-level, no upstream change):

Added QUOTA to the provider's retryableCodes via the profile retryPolicy config, per your recommendation — this survives source upgrades since it lives in ~/.dsh/.
Additionally reordered classifyPiAiError (llm-pi-ai/src/stream.ts) so the 429/rate.?limit check runs before the prose quota classifier, matching the fix already applied to the deepseek adapter's httpErrorCode.
With these, insufficient_quota and 429001 (tpm exhausted) now retry and usually recover within a handful of attempts (observed 8–9 retries auto-recovering).

One residual we'd appreciate guidance on:

A 429001 "inference tpm exhausted" occasionally takes longer to recover than our (10-minute-budgeted) retry window, and we saw a single case where the request gave up at 10/10 (60s delays). Since TPM recovery on a shared relay is governed by aggregate usage and is inherently unpredictable, a purely retry-count-based budget cannot guarantee coverage.
We would welcome a Harness feature where quota/TPM errors on relays are "parked" for a long / unbounded graceful wait (resuming the turn when the quota returns) rather than consuming count-limited retries, or any recommended pattern for handling this class of relay throttling.
Happy to assist with testing or provide more session logs if that helps.

Best regards

[weijiafu14]

Your root-cause holds for the official chain, and that's the right place for it to be fixed. Until it is, there is a parallel path whose retry behavior we have now actually measured, so posting the numbers here in case it unblocks gateway/relay users today.

Through pi2dsh, a transport-carrying Pi provider package rides pi-ai's own transport retry (retryProviderRequest) instead of llm-deepseek's httpErrorCode() + llm-retry. Two properties matter for exactly this thread:

1. 429 is retryable regardless of the body. pi-ai's isRetryableProviderError retries on status === 429 (and ≥500) — there is no insufficient_quota → terminal-QUOTA reclassification in front of it, so a relay's "quota" phrasing on a transient 429 cannot strand the request.
2. Retry-After is honored. Fault-injection against the real pi-ai 0.84.1 transport (crafted wire responses; the OpenAI client + retry code under test are unmodified): a 429 with Retry-After: 2 and an insufficient_quota body waited ~2.0s, retried once, and recovered. Server-requested delays above the 60s default cap fail loudly naming the delay (no silent hang), and the cap itself is caller-raisable (maxRetryDelayMs) — a 429 under a raised cap also recovered in the same run.

Harness and raw verdicts are in the repo: retry-experiment.mjs / retry-experiment-results.json.

Scope honesty: this is a transport-layer verdict (crafted 429/500 responses into the real client/retry code), not an end-to-end run against your specific relay — and it does not fix the official llm-deepseek classification you diagnosed, which still deserves the upstream fix.