fix(providers): retry mid-stream transport drops on the OpenAI-compat wire - #757
Merged
Merged
Conversation
… wire A connection that dies mid-response carries no HTTP status and no salvageable output, so re-issuing is the right call. The Anthropic provider has done this since #747. The OpenAI-compatible provider had no equivalent -- no retry, no attempt loop -- so one dropped chunk ended the whole agent run. Measured cost, terminal-bench 2.1 with deepseek-v4-pro at effort=max (2026-07-27, eval/harbor/jobs/tb21-deepseek-max-1, score 0.449 vs the same harness scoring 0.809 on opus-5): peer closed connection without sending complete message body (incomplete chunked read) ended 8 of 89 trials -- 16% of every failure. regex-chess died this way after 391 seconds of successful work. Identical benchmark, identical error string, identical harness; only the wire differed. What changed ------------------------------------------------------------------ * src/providers/stream_retry.py (new) holds the classifier and its marker list, moved out of anthropic_provider. Dependency-free so openai_compatible can import it without pulling in the Anthropic SDK or creating a cycle. anthropic_provider re-exports it under the original private names, so its call sites and test_stream_watchdog keep resolving. * OpenAICompatibleProvider.chat_stream_response is now a retry wrapper over _stream_attempt (the previous body, unchanged). Retry discipline, mirroring #747: * Bounded at one re-issue. * NEVER retries anything carrying an HTTP status. That guard matters here specifically: the same DeepSeek run has a real image_url 400 (9 trials, a separate bug), and retrying it would just buy a second identical rejection. * Skipped once any text or reasoning has reached the caller, or the transcript would show that prefix twice. Harbor runs --print --output-format stream-json without --include-partial-messages, so both callbacks are None in a scored trial and the retry always fires. * Re-raises the ORIGINAL exception on the last attempt, so callers see the real cause rather than a retry wrapper. Not claimed: that this recovers all 8 trials. One retry covers a blip, not a connection that reliably dies at the same point -- if drops persist through the retry, that points at deepseek-v4-pro's max_output_tokens=8_192 against DeepSeek's documented "reasoning_effort=max needs a longer output window", which is the next thing to check. Tests: 12 new, covering retry-then-succeed, bounded re-raise of the original, status-bearing errors never retried, and both callback paths suppressing the retry. Revert-sensitivity verified: max_attempts=1 fails exactly the two retry tests. Full suite 8853 passed, 0 failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A connection that dies mid-response carries no HTTP status and no salvageable output, so re-issuing is the right call. The Anthropic provider has done this since #747. The OpenAI-compatible provider had no equivalent — no retry, no attempt loop — so one dropped chunk ended the whole agent run.
Measured cost
terminal-bench 2.1,
deepseek-v4-proateffort=max(eval/harbor/jobs/tb21-deepseek-max-1), scoring 0.449 where the same harness scores 0.809 on opus-5:ended 8 of 89 trials — 16% of every failure.
regex-chessdied this way after 391 seconds of successful work. Identical benchmark, identical error string, identical harness; only the wire differed.Failure breakdown of that run for context:
image_urlunsupported (separate bug)What changed
src/providers/stream_retry.py(new) holds the classifier and marker list, moved out ofanthropic_provider. Dependency-free soopenai_compatiblecan import it without pulling in the Anthropic SDK or creating a cycle.anthropic_providerre-exports under the original private names, so its call sites andtest_stream_watchdogkeep resolving.OpenAICompatibleProvider.chat_stream_responseis now a retry wrapper over_stream_attempt(the previous body, unchanged).Retry discipline, mirroring #747:
image_url400 (9 trials, separate bug), and retrying it would buy a second identical rejection.--print --output-format stream-jsonwithout--include-partial-messages, so both callbacks areNonein a scored trial and the retry always fires.Not claimed
That this recovers all 8 trials. One retry covers a blip, not a connection that reliably dies at the same point. If drops persist through the retry, that points at
deepseek-v4-pro'smax_output_tokens=8_192against DeepSeek's documented "reasoning_effort=maxrequires the longer output window" — the next thing to check.Tests
12 new: retry-then-succeed, bounded re-raise of the original, status-bearing errors never retried, both callback paths suppressing the retry, and shared-predicate parity across wires. Revert-sensitivity verified —
max_attempts = 1fails exactly the two retry tests. Full suite 8853 passed, 0 failures.🤖 Generated with Claude Code