Skip to content

fix(llm): add HTTP-status and transport-level retry to Ollama backend#6531

Merged
bug-ops merged 1 commit into
mainfrom
fix/6491-ollama-retry-backoff
Jul 20, 2026
Merged

fix(llm): add HTTP-status and transport-level retry to Ollama backend#6531
bug-ops merged 1 commit into
mainfrom
fix/6491-ollama-retry-backoff

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • Ollama's chat/chat_stream/chat_with_tools sent every request through ollama-rs with no retry, so a single 429/503 or transient connection failure failed the whole turn immediately, unlike Claude/OpenAI/Gemini.
  • ollama-rs discards HTTP status codes on non-2xx responses, so ollama.rs now posts chat requests directly via a dedicated reqwest client, reusing ChatMessageRequest/ChatMessageResponse only for serialization, letting retry::send_with_retry retry on real HTTP 429/503 with Retry-After support — the same helper Claude/OpenAI/Gemini use.
  • A second layer, send_with_transport_retry, retries transient transport failures (timeout, mid-request connection reset) that never reach send_with_retry's status inspection. Connect-phase failures stay unretried and keep RouterProvider's fallback diagnostics (RouterProvider::chat and chat_stream discard last provider error on fallback exhaustion #5821) intact.
  • Also fixes a UTF-8 chunk-boundary data-loss bug in the streaming NDJSON parser this change introduces: it now buffers raw bytes across chunks and decodes only once a complete line is assembled, instead of decoding each chunk independently and silently dropping one that splits a multi-byte character.

Closes #6491

Test plan

  • cargo +nightly fmt --check
  • cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings
  • cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins
  • RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" cargo doc --no-deps --workspace --features "desktop,ide,server,chat,pdf,scheduler"
  • LLM Serialization Gate: ran live against a real local Ollama instance (0.32.1, qwen2.5:7b), including tool-call round trips; all previously-ignored integration tests plus a new chat_with_tools integration test pass
  • New unit tests: HTTP 429/503+Retry-After retry through the real provider methods (wiremock), transport-level retry, UTF-8 chunk-boundary NDJSON buffering, stream-flag JSON body correctness, chat_with_tools retry

Ollama's chat/chat_stream/chat_with_tools sent every request through
ollama-rs with no retry, so a single 429/503 or transient connection
failure failed the whole turn immediately, unlike Claude/OpenAI/Gemini.
ollama-rs discards HTTP status codes on non-2xx responses, so ollama.rs
now posts chat requests directly via a dedicated reqwest client,
reusing ChatMessageRequest/ChatMessageResponse only for serialization,
letting retry::send_with_retry retry on real 429/503 with Retry-After
support. A second layer, send_with_transport_retry, retries transient
transport failures (timeout, mid-request connection reset) that never
reach send_with_retry's status inspection. Connect-phase failures stay
unretried and keep RouterProvider's fallback diagnostics intact.

Also fixes a UTF-8 chunk-boundary data-loss bug in the streaming NDJSON
parser this change introduces: it now buffers raw bytes across chunks
and decodes only once a complete line is assembled, instead of
decoding each chunk independently and silently dropping one that
splits a multi-byte character.

Closes #6491
@bug-ops
bug-ops enabled auto-merge (squash) July 20, 2026 00:38
@github-actions github-actions Bot added documentation Improvements or additions to documentation llm zeph-llm crate (Ollama, Claude) rust Rust code changes dependencies Dependency updates bug Something isn't working size/XL Extra large PR (500+ lines) labels Jul 20, 2026
@bug-ops
bug-ops merged commit 555309b into main Jul 20, 2026
122 of 131 checks passed
@bug-ops
bug-ops deleted the fix/6491-ollama-retry-backoff branch July 20, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dependencies Dependency updates documentation Improvements or additions to documentation llm zeph-llm crate (Ollama, Claude) rust Rust code changes size/XL Extra large PR (500+ lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

zeph-llm: Ollama backend has no rate-limit/transient-failure retry, unlike Claude/OpenAI/Gemini

1 participant