fix(ai-proxy): preserve upstream 429/5xx error response body#13565
Merged
Conversation
ai-proxy and ai-proxy-multi returned only the status code and closed the connection before reading the body for 429/5xx responses, so the provider's error details were discarded and never reached the client or the logs. Now the error body is read before closing: ai-proxy-multi logs it when it falls back to another instance (the failed attempt's body is otherwise lost), and it is returned to the client together with the upstream status code and Content-Type when no retry happens.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR ensures ai-proxy / ai-proxy-multi preserve upstream provider error details by reading and propagating 429/5xx response bodies (to clients when not retried, and to logs when falling back), improving diagnosability of upstream failures.
Changes:
- Read and return upstream
429/5xxresponse bodies and preserve upstreamContent-Type. - Log upstream error bodies in
ai-proxy-multiwhen falling back to another instance. - Add/adjust tests and update plugin documentation to describe the new error-body behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
apisix/plugins/ai-proxy/base.lua |
Reads upstream error bodies and returns them (plus forwards Content-Type), and passes error body into retry callback. |
apisix/plugins/ai-proxy-multi.lua |
Logs upstream error body on fallback and wires error body into retry handler. |
t/plugin/ai-proxy.t |
Adds test coverage asserting upstream 5xx error bodies are returned to clients. |
t/plugin/ai-proxy-multi-retry.t |
Extends retry tests to assert error body is logged on fallback and returned when not retried. |
docs/en/latest/plugins/ai-proxy.md |
Documents upstream error response behavior and body preservation. |
docs/en/latest/plugins/ai-proxy-multi.md |
Documents upstream error response behavior for fallback vs. no-retry cases. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
shreemaan-abhishek
approved these changes
Jun 17, 2026
AlinsRan
approved these changes
Jun 17, 2026
membphis
approved these changes
Jun 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When an upstream LLM provider returns
429or a5xx,ai-proxy/ai-proxy-multireturned only the status code and closed the connection before reading the body. The provider's error payload (rate-limit details, validation errors, etc.) was discarded — the client got an empty body and nothing was logged, which makes upstream failures hard to diagnose.This reads the upstream error body before closing the connection and routes it to where it is useful:
ai-proxy-multilogs the error body when it falls back to another instance, since that failed attempt's body never reaches the client (a later attempt responds instead).ai-proxy, no matchingfallback_strategy,max_retriesexhausted, or the failure took longer thanretry_on_failure_within_ms), the upstream status code and error body are returned to the client, preserving the upstreamContent-Type.Error bodies are small, so the body is read with a single
read_body(); no extra config is introduced.Fixes #13501
Checklist