Skip to content

fix(ai-proxy): preserve upstream 429/5xx error response body#13565

Merged
nic-6443 merged 2 commits into
apache:masterfrom
nic-6443:fix/ai-proxy-13501-error-body
Jun 17, 2026
Merged

fix(ai-proxy): preserve upstream 429/5xx error response body#13565
nic-6443 merged 2 commits into
apache:masterfrom
nic-6443:fix/ai-proxy-13501-error-body

Conversation

@nic-6443

Copy link
Copy Markdown
Member

Description

When an upstream LLM provider returns 429 or a 5xx, ai-proxy / ai-proxy-multi returned only the status code and closed the connection before reading the body. The provider's error payload (rate-limit details, validation errors, etc.) was discarded — the client got an empty body and nothing was logged, which makes upstream failures hard to diagnose.

This reads the upstream error body before closing the connection and routes it to where it is useful:

  • ai-proxy-multi logs the error body when it falls back to another instance, since that failed attempt's body never reaches the client (a later attempt responds instead).
  • When the request is not retried (single-instance ai-proxy, no matching fallback_strategy, max_retries exhausted, or the failure took longer than retry_on_failure_within_ms), the upstream status code and error body are returned to the client, preserving the upstream Content-Type.

Error bodies are small, so the body is read with a single read_body(); no extra config is introduced.

Fixes #13501

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

ai-proxy and ai-proxy-multi returned only the status code and closed the
connection before reading the body for 429/5xx responses, so the provider's
error details were discarded and never reached the client or the logs.

Now the error body is read before closing: ai-proxy-multi logs it when it
falls back to another instance (the failed attempt's body is otherwise lost),
and it is returned to the client together with the upstream status code and
Content-Type when no retry happens.
Copilot AI review requested due to automatic review settings June 16, 2026 09:35
@dosubot dosubot Bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Jun 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR ensures ai-proxy / ai-proxy-multi preserve upstream provider error details by reading and propagating 429/5xx response bodies (to clients when not retried, and to logs when falling back), improving diagnosability of upstream failures.

Changes:

  • Read and return upstream 429/5xx response bodies and preserve upstream Content-Type.
  • Log upstream error bodies in ai-proxy-multi when falling back to another instance.
  • Add/adjust tests and update plugin documentation to describe the new error-body behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
apisix/plugins/ai-proxy/base.lua Reads upstream error bodies and returns them (plus forwards Content-Type), and passes error body into retry callback.
apisix/plugins/ai-proxy-multi.lua Logs upstream error body on fallback and wires error body into retry handler.
t/plugin/ai-proxy.t Adds test coverage asserting upstream 5xx error bodies are returned to clients.
t/plugin/ai-proxy-multi-retry.t Extends retry tests to assert error body is logged on fallback and returned when not retried.
docs/en/latest/plugins/ai-proxy.md Documents upstream error response behavior and body preservation.
docs/en/latest/plugins/ai-proxy-multi.md Documents upstream error response behavior for fallback vs. no-retry cases.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/plugins/ai-proxy/base.lua
Comment thread apisix/plugins/ai-proxy/base.lua
Comment thread apisix/plugins/ai-proxy-multi.lua
Comment thread t/plugin/ai-proxy.t
Comment thread t/plugin/ai-proxy.t
@nic-6443 nic-6443 merged commit 93ace5d into apache:master Jun 17, 2026
37 checks passed
@nic-6443 nic-6443 deleted the fix/ai-proxy-13501-error-body branch June 17, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: upstream error response body is discarded for 429/5xx responses

5 participants