[Bug] Spring Cloud Gateway 4.3.3 plugin may inject a stale sw8 context from a previous request

[15000854736]

Describe the bug

Under concurrent traffic, the SkyWalking Spring Cloud Gateway 4.x plugin may propagate a stale sw8 context that belongs to a previous request.

The downstream service confirms that the sw8 header exists, but the Trace ID activated by the downstream agent is not the Trace ID of the current gateway request. It is an older Trace ID previously created by the same gateway instance.

This is different from a missing-header problem: propagation happens, but the propagated context appears stale.

Environment

• SkyWalking Java Agent: 9.6.0 (Java 25 image)
• JDK: 25
• Spring Cloud Gateway: 4.3.3
• Spring WebFlux: 6.2.19
• Reactor Netty: 1.2.18
• Reactor Core: 3.7.19
• Runtime: Kubernetes/Linux
• Gateway plugin: apm-spring-cloud-gateway-4.x-plugin-9.6.0.jar

The standalone optional apm-spring-webflux-6.x-plugin and apm-netty-http-4.1.x-plugin are not enabled, so the Gateway plugin is the only plugin responsible for outbound Gateway trace propagation.

Evidence

For one request, the gateway reports the current trace as:

gateway current traceId:
71c8b80380e549359024db364b126704.328.17898163386063873

The downstream service receives sw8, but activates another trace:

downstream: sw8 exists: true
downstream active traceId:
71c8b80380e549359024db364b126704.328.17898155067262141

Both IDs have the same gateway process/instance prefix, while the timestamp component of the downstream Trace ID is about 833 seconds older. Repeated requests show the same pattern: the gateway creates unique current Trace IDs, but the downstream service sometimes activates a historical gateway Trace ID.

Steps to reproduce

1. Run a Spring Cloud Gateway 4.3.3 application with SkyWalking Java Agent 9.6.0 and enable apm-spring-cloud-gateway-4.x-plugin-9.6.0.jar.
2. Route requests to another Spring Boot service that also runs the SkyWalking agent.
3. Keep the standalone WebFlux and Netty HTTP optional plugins disabled.
4. Send concurrent requests through the gateway to the downstream service.
5. Log the gateway's current Trace ID and, in the downstream service, log both whether sw8 exists and the activated Trace ID.
6. Compare the gateway and downstream Trace IDs for each request.

Expected behavior

The downstream service should join the trace of the current gateway request.

Actual behavior

The downstream service receives sw8, but intermittently joins a trace created by an earlier request on the same gateway instance.

Suspected cause

In the v412x implementation, NettyRoutingGetHttpClientV412Interceptor.afterMethod stores a request-scoped ContextSnapshot in the dynamic field of the HttpClient returned by NettyRoutingFilter#getHttpClient:

EnhancedInstance retEnhancedInstance = (EnhancedInstance) ret;
retEnhancedInstance.setSkyWalkingDynamicField(retEnhanceObjectCache);

That Reactor Netty HttpClient may be shared/reused across requests. If multiple requests write their ContextSnapshot to the same enhanced object, a later duplicate/connect/send stage may observe a snapshot from another request and inject stale sw8 data.

The following local experiment prevents the shared-object write by deriving a request-local HttpClient before attaching the snapshot:

Object requestHttpClient = ret instanceof HttpClient
    ? ((HttpClient) ret).doOnRequest((request, connection) -> {})
    : ret;

if (requestHttpClient instanceof EnhancedInstance && ContextManager.isActive()) {
    ContextSnapshot contextSnapshot = ContextManager.capture();
    EnhanceObjectCache cache = new EnhanceObjectCache();
    cache.setContextSnapshot(contextSnapshot);
    ((EnhancedInstance) requestHttpClient).setSkyWalkingDynamicField(cache);
}
return requestHttpClient;

A regression test also confirms that two invocations using the same shared HttpClient return two distinct derived instances.

Could the maintainers please confirm whether storing the request ContextSnapshot on the shared HttpClient is expected, and whether the context should instead be attached to a request-local object or captured closer to the actual send operation?

Additional context

• Downgrading the application from Spring Cloud Gateway 4.3.5 to 4.3.3 did not resolve the stale propagation.
• Gateway request Trace IDs themselves remain unique; the mismatch appears specifically at the outbound propagation boundary.
• No raw sw8 values are included because they contain encoded service and endpoint metadata. Sanitized diagnostic logs can be provided if needed.

[wu-sheng]

Thanks for the analysis — it is correct, and the level of detail made this quick to confirm.

I verified it against Spring Cloud Gateway 4.3.3 and reactor-netty 1.2.18 sources. NettyRoutingFilter#getHttpClient returns the shared this.httpClient bean itself unless a connect timeout is configured, so the plugin was parking per-request state on a per-JVM object. The window is also wider than a tight race: getHttpClientMono evaluates getHttpClient eagerly inside filter(), while the read (HttpClientConnect#duplicate through HttpClient#headers) happens inside flatMapMany at subscription time, after filter() has returned. Any request reaching getHttpClient in between overwrites the snapshot.

The fix is in apache/skywalking-java#830, built on your proposal. Four things are different from your draft, and it is worth saying why.

1. headers(h -> { }) instead of doOnRequest((req, conn) -> { }) to force the copy.

Both go through the same duplicate(), so both fix the bug. But doOnRequest has a side effect on every gateway user. HttpClientConfig#defaultConnectionObserver takes a fast path only when all of the doOn* callbacks are null:

if (doAfterRequest == null && doAfterResponseSuccess == null && doOnRedirect == null &&
        doOnRequest == null && doOnRequestError == null && doOnResponse == null && doOnResponseError == null) {
    return super.defaultConnectionObserver();
}
else {
    return super.defaultConnectionObserver().then(new HttpClientDoOn(...));
}

Installing a no-op doOnRequest moves every request onto the composed-observer path and invokes the callback on each REQUEST_PREPARED. headers(h -> { }) only copies the header map, which Spring Cloud Gateway copies again one line later, so the chain — which already duplicates on headers(), request() and uri() — gains one duplication and no observer. To be explicit for anyone reading this later: the consumer is applied to configuration().headers.copy(), so nothing is reset; the derived client's headers are value-identical to the shared bean's.

2. The ContextManager.isActive() check runs first. In the draft the client is derived before the check, so an untraced request pays for the duplication, and HttpClientConnectDuplicateV412Interceptor copies the shared bean's current cache onto the derived client which then never gets overwritten. Checking first means untraced requests allocate nothing and cannot inherit anything.

3. If the derived object is not an EnhancedInstance, the original is returned rather than a derived client that carries no snapshot, so propagation never fails silently.

4. The regression test asserts the behaviour rather than reactor-netty's contract. NettyRoutingGetHttpClientV412InterceptorTest now asserts that the shared client is never stamped and that two requests get distinct snapshots. Both assertions fail against the current interceptor and pass with the patch.

Could you verify it locally?

git clone --recurse-submodules https://github.com/apache/skywalking-java.git
cd skywalking-java
git checkout fix/gateway-4.x-per-request-context-snapshot
./mvnw clean package -Dmaven.test.skip=true

The agent is in skywalking-agent/; copy optional-plugins/apm-spring-cloud-gateway-4.x-plugin-*.jar into plugins/ as you do today, and keep the standalone WebFlux and Netty HTTP plugins disabled so the comparison is clean.

Two questions, because one part of your evidence does not fit this root cause.

The stale trace is about 832 seconds old (1789816338606 vs 1789815506726 ms), and both ids carry the same thread component .328.. A concurrent overwrite hands you a sibling request's trace from milliseconds ago, usually from another event loop — not one from fourteen minutes ago on the same thread. So there may be a second cause.

1. Is spring.cloud.gateway.httpclient.connect-timeout, or per-route connect-timeout metadata, set in your configuration? If it is, getHttpClient returns this.httpClient.option(...), option() duplicates, and you were already getting a fresh client per request — which would mean the shared-bean race never applied to you and this patch will not change what you see.

2. If the stale sw8 survives the patch, the next suspect is a leaked TracingContext on the event loop thread. TraceSegment#relatedGlobalTrace only replaces the trace id while it is still a NewDistributedTraceId, so a context that leaked and is reused on that thread keeps emitting the trace id it was born with — which matches an old id on a matching thread far better than a race does. Sanitized logs pairing the gateway's current trace id with the thread name would tell us.

Either way this is a real cross-request context bleed and worth fixing on its own. Thanks again for the report and the patch.

[wu-sheng]

@15000854736 Any update? I am wondering whether I should merge the PR?