[Bug] -p batch mode hangs on a 429 with a long reset; unknown settings keys are silently accepted

[thomaswillner]

Summary

Affected area: settings validation, provider retry/backoff, and non-interactive (-p) batch runs.

Two findings from one incident. They are separable, but they compound: the settings key I expected to provide failover turned out to be inert, so a single provider cooldown had nothing to fall back to — and instead of failing, the batch run hung until I killed it.

This is closely related to #1405 (terminal quota response leaves the TUI on Waiting). That report covers OAuth subscription exhaustion in the interactive TUI; what I hit is the API-key 429-with-long-reset variant in -p batch mode. The user-visible rule @sankks11 proposed there — a terminal limit should end the request and surface any reset information — would cover my case too, so these may share a fix.

Environment

• prime-agent 0.7.2 (npm global)
• macOS arm64
• Models served through a local OpenAI-compatible proxy (api: openai-completions, custom models.json providers)

Finding 1 — unknown settings keys are silently accepted

settings.json contained a fallbackModels array. It validated cleanly, emitted no warning, and did nothing.

The key appears 0 times in the shipped dist/bundle. Control greps against the same bundle, to show the method finds keys that do exist:

key	occurrences in dist/bundle
fallbackModels	0
rlmMaxDepth	47
autoRefine	75
steeringMode	22

Because unknown keys are accepted without complaint, a harness can appear failover-protected while having no failover at all. The same silence would swallow a typo, or a key removed in a later release — the config keeps validating and the behavior quietly disappears.

Suggestion: reject unknown settings keys, or warn loudly on load. Silent acceptance turns config drift into invisible no-ops with operational consequences.

Finding 2 — a 429 with a long advertised reset hangs batch mode

The sole configured default model began returning HTTP 429 with a long advertised reset (~102 hours — a provider-family cooldown).

A prime-agent -p "..." batch run then produced zero output and never exited. I killed it externally after 240s. The daemon log ends at:

Supervisor command prompt_and_wait failed: Error: Daemon worker client closed

retry.provider.maxRetryDelayMs was 30000. Per the settings documentation, a server-requested delay above that cap should "fail immediately with an informative error" — but the run hung instead of surfacing a terminal error.

Control: the same run with --provider/--model pointed at a healthy route exited 0 immediately. The failure is isolated to the dead route plus the absent failover, not to the prompt or the harness invocation.

Suggestion: treat a retry delay above retry.provider.maxRetryDelayMs as terminal in batch mode as well as interactive — a 429 carrying an hours-long reset should end the turn with a structured failure and a non-zero exit, not a silent hang. A hang is materially worse than an error for unattended runs: it consumes the whole wall-clock budget and produces nothing to act on.

Related feature interest

Ordered, health-gated fallbackModels across configured providers would let long-running and autonomous work survive a provider cooldown without an external wrapper. I have kept that out of the two bug findings above deliberately — happy to move it to Feature requests if that is the better home for it.

I am currently working around this with an external wrapper that health-probes a cross-provider chain before and between attempts.

Offer

I have a working patch for the failover piece on a fork branch (thomaswillner/prime-agent@eb6f1ab, 19 files). I am not requesting that it be reviewed or merged — I understand from CONTRIBUTING.md that implementation happens on maintainer invitation, and I opened it before reading that, which was my mistake. Flagging it only so the work is visible if it is useful.

Happy to supply the full reproduction evidence — bundle greps, daemon logs, proxy responses — or to re-test any proposed fix against the same 429 route.

[thomaswillner]

Follow-up: I traced finding 2 and it has a concrete mechanism, which also corrects how I framed the request above.

The cap is not implemented. retry.provider.maxRetryDelayMs is declared, forwarded, defaulted and documented, but nothing consumes it. Repository-wide search on current main returns 10 files — types.ts (declaration), simple-options.ts and proxy.ts (forwarding), sdk.ts and settings-manager.ts (default 60000, plus migration of the older retry.maxDelayMs), side-question.ts, agent.ts, both CHANGELOGs and settings.md. No provider implementation is among them. Independently, the shipped 0.7.2 dist/bundle contains the token 22 times with zero comparisons — every site is an assignment or a forward.

What happens instead. The vendored Stainless SDKs — anthropic-6NOSCFKS.js and chunk-ENTVCTJV.js, the latter being the OpenAI SDK required by openai-completions-7BADVXTD.js, which is the path in my report — both do:

const retryAfterHeader = responseHeaders?.get("retry-after");
if (retryAfterHeader && !timeoutMillis) {
  const timeoutSeconds = parseFloat(retryAfterHeader);
  if (!Number.isNaN(timeoutSeconds)) timeoutMillis = timeoutSeconds * 1e3;
}
...
await sleep(timeoutMillis);
return this.makeRequest(options, retriesRemaining - 1, requestLogID);

Unbounded, at maxRetries ?? 2.

The number. Re-measured today against the still-cooled route:

HTTP/1.1 429 Too Many Requests
Retry-After: 153971

Seconds — so sleep(153_971_000), 42.8 hours, up to twice. During the original incident the advertised reset was ~102h47m, i.e. ~4.28 days. That is the whole of the "zero output and never exits" behaviour: the process is asleep inside the SDK. It also explains why raising the cap from 30000 to 60000 changed nothing for me.

Correcting my own request 2. I asked for the cap to be treated as terminal in batch mode too, which implied it worked interactively and merely missed the headless path. That was wrong — it is not enforced anywhere, for any provider or mode. The accurate request is: implement the cap at the point where a server-requested delay is turned into a sleep, and fail with an informative error above it, as settings.md already documents and as packages/ai/CHANGELOG.md 0.50.8 announced when the option was added (citing badlogic/pi-mono#1123, trigger example "e.g., Google Gemini CLI").

Requests 1 (reject unknown settings keys) and 3 (ordered fallbackModels) are unchanged. Request 1 gains weight from this: a documented option that no code reads is the same class of silent no-op as an unknown key, just one the schema cannot catch.

I have posted the same evidence on #1405, where it distinguishes two mechanisms converging on one symptom: this returned-429 sleep, and the unbounded stream read that report describes for OAuth exhaustion where no Retry-After is returned at all.

Caveat on method: this comes from reading the installed 0.7.2 bundle plus main via code search. I have not attached a debugger to a live 429, so the sleep is inferred from the code path and the measured header rather than observed in a stack trace.