Route until a usable upstream response

[gitcommit90]

Summary

• continue named routes and OAuth account pools after every non-2xx response
• reject immediate stream errors, empty streams, invalid JSON, and explicit 2xx error payloads before selecting a route member
• keep failure classification limited to cooldowns and diagnostics

Tests

• npm test (300 passed)
• git diff --check

---

[devin-ai-integration]

Devin Review found 1 potential issue.

View 1 additional finding in Devin Review.

[devin-ai-integration]

🟡 Slow reasoning responses can be buffered whole and rescanned repeatedly, bogging down the gateway

The entire accumulated body of a streaming response is re-scanned from the start (hasProductiveResponsesEvent(text) at src/lib/router.js:615) on every incoming network chunk with no size limit, before any bytes are forwarded to the client, so a long non-output preamble makes the scanning work grow quadratically and holds all of it in memory.
Impact: A reasoning-heavy or slow upstream response can spike CPU and memory and delay the client from seeing anything until real output finally begins.

Unbounded accumulation after removing the inspection byte cap

Previously inspectEarlyResponsesSse took maxBytes = 64 * 1024 and looped while (bytes < maxBytes), so at most 64 KiB was buffered/inspected before it fell back to rebuildResponseWithPrelude and streamed the rest live. This PR removed the parameter and changed the loop to while (true) (src/lib/router.js:602). Now the loop keeps reading until a recognized productive event appears or the stream ends.

Within each iteration text += decoder.decode(...) accumulates the whole body (src/lib/router.js:607), and both parseEarlyResponsesFailure(text, response) (src/lib/router.js:608) and hasProductiveResponsesEvent(text) (src/lib/router.js:615) re-split and re-JSON.parse the ENTIRE accumulated text every chunk. For a response that emits a large non-productive preamble (e.g. long reasoning summaries on the Responses/Codex surface, which are not counted as productive events), this is O(total_size^2) synchronous parsing on the event loop plus retaining every chunk in the chunks array. Normal streams that emit content in the first chunk break immediately and are unaffected; the regression is specific to large slow preambles. Note the sibling passthrough pipeOpenAiCompatibleSse still caps its buffer at 128 KiB (src/lib/router.js:531), highlighting the missing bound here.

(Refers to lines 602-616)

Prompt for agents

inspectEarlyResponsesSse in src/lib/router.js previously bounded its pre-output inspection with a maxBytes cap (default 64 KiB) via `while (bytes < maxBytes)`. This PR removed the cap and now loops `while (true)`, accumulating the full response text and re-running parseEarlyResponsesFailure(text) and hasProductiveResponsesEvent(text) over the entire accumulated string on every chunk. For upstream responses that emit a large non-productive preamble before real output (notably long reasoning summaries on the Responses/Codex surface, which are not treated as productive events), this causes quadratic re-parsing on the event loop and unbounded memory buffering of all chunks. Consider reintroducing a byte/size bound after which the response falls back to rebuildResponseWithPrelude and streams live, and/or only scanning newly appended text rather than re-parsing the whole accumulated buffer each iteration. Ensure the timeout (memberSignal) interaction is considered, since the request timeout stays armed for the whole inspection window.

---

Was this helpful? React with 👍 or 👎 to provide feedback.