Skip to content

[fix][proxy] Avoid intermittent 502 when admin proxy follows a broker redirect for a request with a body#25919

Merged
lhotari merged 1 commit into
apache:masterfrom
lhotari:lh-fix-proxy-redirect-502
Jun 3, 2026
Merged

[fix][proxy] Avoid intermittent 502 when admin proxy follows a broker redirect for a request with a body#25919
lhotari merged 1 commit into
apache:masterfrom
lhotari:lh-fix-proxy-redirect-502

Conversation

@lhotari
Copy link
Copy Markdown
Member

@lhotari lhotari commented Jun 2, 2026

Motivation

AdminProxyHandler (the Pulsar admin proxy) follows broker HTTP 307 redirects internally: when the proxy forwards an admin request to a broker that does not own the topic's bundle, the broker responds with 307 and the proxy follows the redirect to the owner broker, replaying the request body.

Intermittently, admin requests that carry a body (e.g. PUT createNonPartitionedTopic, POST setDelayedDeliveryPolicy) fail through the proxy with HTTP 502 Bad Gateway. The failure is timing-dependent and surfaces as flakiness in ProxyRedirectTest.testProxyHandlesReplayingContent, more frequently on CPU-constrained environments such as CI runners. Requests without a body (GET) are unaffected.

Root cause: Jetty's RedirectProtocolHandler.onSuccess(Response) aborts the in-flight request with HttpRequestException: "Aborting request after receiving a 307 response" whenever the request still has a body to send (request.getBody() != null), in order to stop streaming the body:

public void onSuccess(Response response) {
    // The request may still be sending content, stop it.
    Request request = response.getRequest();
    if (request.getBody() != null)
        request.abort(new HttpRequestException("Aborting request after receiving a %d response"
            .formatted(response.getStatus()), request));
}

When the broker returns the 307 before the proxy has finished sending the request body, that abort races ahead of the redirect continuation (onComplete) and is delivered to the proxy as the request failure. AbstractProxyServlet.onProxyResponseFailure then maps it (a non-TimeoutException) to HTTP 502 Bad Gateway. The cause is logged by Jetty only at DEBUG, which is why it is invisible in normal proxy logs.

Modifications

  • Add NonAbortingRedirectProtocolHandler, a RedirectProtocolHandler subclass whose onSuccess does not abort the in-flight request.
  • Register it in AdminProxyHandler#customizeHttpClient in place of the stock RedirectProtocolHandler.

The redirect is still driven by RedirectProtocolHandler#onComplete from the response status and Location header, so redirects are followed exactly as before (the body is replayed by the existing ReplayableProxyContentProvider); only the abort that produced the spurious 502 is removed.

Verifying this change

This change is already covered by existing tests, specifically ProxyRedirectTest.testProxyHandlesReplayingContent, which exercises the proxy redirect + request-body-replay path.

It was additionally validated against a reliable local reproduction: with the test's topic loop raised to 5000 and run in a single-CPU container, the 502 reproduced within ~1–3 minutes before this change, and passed 5/5 consecutive runs after it (with no regression to a 504/idle-timeout path).

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

… redirect for a request with a body

The admin proxy (AdminProxyHandler) follows broker HTTP 307 redirects internally and
replays the request body to the bundle-owner broker. Jetty's
RedirectProtocolHandler.onSuccess aborts the in-flight request with
HttpRequestException ("Aborting request after receiving a 307 response") whenever the
request still has a body to send. When the broker returns the 307 before the proxy has
finished streaming the body, that abort races ahead of the redirect continuation
(onComplete) and is delivered to the proxy as the request failure, which
AbstractProxyServlet maps (non-TimeoutException) to a spurious HTTP 502 Bad Gateway.
Requests without a body (GET) are unaffected.

Register a RedirectProtocolHandler subclass whose onSuccess does not abort. The redirect
is still driven by onComplete from the response status and Location header, so redirects
are followed exactly as before (the body is replayed by ReplayableProxyContentProvider);
only the abort that produced the 502 is removed.

Assisted-by: Claude Code
Copy link
Copy Markdown
Contributor

@void-ptr974 void-ptr974 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 3, 2026
@lhotari lhotari merged commit 2acee32 into apache:master Jun 3, 2026
44 checks passed
lhotari added a commit that referenced this pull request Jun 3, 2026
… redirect for a request with a body (#25919)

(cherry picked from commit 2acee32)
lhotari added a commit that referenced this pull request Jun 3, 2026
… redirect for a request with a body (#25919)

(cherry picked from commit 2acee32)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants