[fix][proxy] Avoid intermittent 502 when admin proxy follows a broker redirect for a request with a body#25919
Merged
Conversation
… redirect for a request with a body
The admin proxy (AdminProxyHandler) follows broker HTTP 307 redirects internally and
replays the request body to the bundle-owner broker. Jetty's
RedirectProtocolHandler.onSuccess aborts the in-flight request with
HttpRequestException ("Aborting request after receiving a 307 response") whenever the
request still has a body to send. When the broker returns the 307 before the proxy has
finished streaming the body, that abort races ahead of the redirect continuation
(onComplete) and is delivered to the proxy as the request failure, which
AbstractProxyServlet maps (non-TimeoutException) to a spurious HTTP 502 Bad Gateway.
Requests without a body (GET) are unaffected.
Register a RedirectProtocolHandler subclass whose onSuccess does not abort. The redirect
is still driven by onComplete from the response status and Location header, so redirects
are followed exactly as before (the body is replayed by ReplayableProxyContentProvider);
only the abort that produced the 502 is removed.
Assisted-by: Claude Code
nodece
approved these changes
Jun 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
AdminProxyHandler(the Pulsar admin proxy) follows broker HTTP307redirects internally: when the proxy forwards an admin request to a broker that does not own the topic's bundle, the broker responds with307and the proxy follows the redirect to the owner broker, replaying the request body.Intermittently, admin requests that carry a body (e.g.
PUT createNonPartitionedTopic,POST setDelayedDeliveryPolicy) fail through the proxy with HTTP 502 Bad Gateway. The failure is timing-dependent and surfaces as flakiness inProxyRedirectTest.testProxyHandlesReplayingContent, more frequently on CPU-constrained environments such as CI runners. Requests without a body (GET) are unaffected.Root cause: Jetty's
RedirectProtocolHandler.onSuccess(Response)aborts the in-flight request withHttpRequestException: "Aborting request after receiving a 307 response"whenever the request still has a body to send (request.getBody() != null), in order to stop streaming the body:When the broker returns the
307before the proxy has finished sending the request body, that abort races ahead of the redirect continuation (onComplete) and is delivered to the proxy as the request failure.AbstractProxyServlet.onProxyResponseFailurethen maps it (a non-TimeoutException) to HTTP 502 Bad Gateway. The cause is logged by Jetty only atDEBUG, which is why it is invisible in normal proxy logs.Modifications
NonAbortingRedirectProtocolHandler, aRedirectProtocolHandlersubclass whoseonSuccessdoes not abort the in-flight request.AdminProxyHandler#customizeHttpClientin place of the stockRedirectProtocolHandler.The redirect is still driven by
RedirectProtocolHandler#onCompletefrom the response status andLocationheader, so redirects are followed exactly as before (the body is replayed by the existingReplayableProxyContentProvider); only the abort that produced the spurious 502 is removed.Verifying this change
This change is already covered by existing tests, specifically
ProxyRedirectTest.testProxyHandlesReplayingContent, which exercises the proxy redirect + request-body-replay path.It was additionally validated against a reliable local reproduction: with the test's topic loop raised to 5000 and run in a single-CPU container, the 502 reproduced within ~1–3 minutes before this change, and passed 5/5 consecutive runs after it (with no regression to a
504/idle-timeout path).Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes