Memory Leak in WebFlux 3.0 + micrometer-tracing-bridge-brave + enableAutomaticContextPropagation

[davidmelia]

Hi,

In production, I've upgraded from spring boot webflux + zipkin 2.7.x App to spring boot webflux + zipkin 3.0.4 and I have noticed a memory leak in the heap.

I've distilled the problem down and there is a memory leak with the combination of WebFlux 3.0 + micrometer-tracing-bridge-brave + enableAutomaticContextPropagation. https://github.com/davidmelia/spring-cloud-function-zipkin/tree/memory_leak shows an example. If you run the app and ping the health check then over time the heap fills

#!/bin/bash
for i in {1..500}
do
   curl http://localhost:8080/actuator/health
done

You will notice brave.baggage.CorrelationUpdateScope$Multiple is never totally reclaimed by garbage collection and builds up over time:

I've plugged the heap dump into the eclipse memory analyser and it looks like reactor.netty.resources.DefaultLoopResources$EventLoop is keeping hold of thread locals:

and drilling into the thread local there is a huge array of brave.baggage.CorrelationUpdateScope$Multiple

N.B I appreciate this looks like a problem with either Hooks.enableAutomaticContextPropagation or micrometer-tracing-bridge-brave however it manifest in Spring Boot Webflux

(relates to #34201)

[marcingrzejszczak]

Should be fixed via micrometer-metrics/tracing#213. Can you check if your problem is gone with 1.0.4-SNAPSHOT of tracing? (I did check that manually but wanted you to confirm that too)

[davidmelia]

I've upgraded to 1.0.4-SNAPSHOT (https://github.com/davidmelia/spring-cloud-function-zipkin/blob/memory_leak_potential_fix/pom.xml) but get a different issue

	*__checkpoint ⇢ HTTP GET "/actuator/health" [ExceptionHandlingWebHandler]
Original Stack Trace:
		at io.micrometer.tracing.handler.TracingObservationHandler.getRequiredSpan(TracingObservationHandler.java:183) ~[micrometer-tracing-1.0.4-SNAPSHOT.jar:1.0.4-SNAPSHOT]
		at io.micrometer.tracing.handler.PropagatingReceiverTracingObservationHandler.onStop(PropagatingReceiverTracingObservationHandler.java:97) ~[micrometer-tracing-1.0.4-SNAPSHOT.jar:1.0.4-SNAPSHOT]

Hmmmmm ...

[marcingrzejszczak]

Wow, I see that too now (I had no-error case for sure in the meantime) - let me look into that

[marcingrzejszczak]

OK I found a bug, I've just applied a fix in tracing (micrometer-metrics/tracing@b8c4ae6) so wait a couple of minutes for the new snapshots

[davidmelia]

@marcingrzejszczak update came down fine

and I get the memory leak:

My project is https://github.com/davidmelia/spring-cloud-function-zipkin/blob/memory_leak_potential_fix/pom.xml and ran the below a few times

#!/bin/bash
for i in {1..500}
do
   curl http://localhost:8080/actuator/health
done

using visualvm memory sampler

[marcingrzejszczak]

wow, that's strange, previously if you ran your app with -ea you would have exceptions being thrown and that was fixed, I thought that that is the root cause. I'll need to investigate this further then

[davidmelia]

Sorry @marcingrzejszczak to be clear when I say same issue it now does work (i.e. no stack trace) but I get the original memory leak.

[marcingrzejszczak]

I have a workaround to get rid of memory leaks. Set this property

management:
  tracing:
    baggage:
      correlation:
        enabled: false

and set this bean

@Bean
  @Order(1)
  CorrelationScopeCustomizer myCorrelationFieldsCorrelationScopeCustomizer(TracingProperties tracingProperties) {
    return (builder) -> {
      List<String> correlationFields = tracingProperties.getBaggage().getCorrelation().getFields();
      for (String field : correlationFields) {
        builder.add(CorrelationScopeConfig.SingleCorrelationField.newBuilder(BaggageField.create(field))
                .build());
      }
    };
  }

in essence it removes the flush on update feature from Brave that is causing the memory leak. We're still investigating the whole problem.

BTW @wilkinsona I think this bug is applicable to all Webflux projects and we should disable the flush on update feature by default for now

[wilkinsona]

Thanks, @marcingrzejszczak. We've got approximately a month till the next Boot releases when we could do that. Perhaps that'll give us some time to figure out a different solution.

[davidmelia]

Thanks @marcingrzejszczak - have pushed your recommendations to https://github.com/davidmelia/spring-cloud-function-zipkin/blob/memory_leak_potential_fix/pom.xml and memory leak has gone.

One slight amendment (I assume a typo) is you need

management:
  tracing:
    baggage:
      correlation:
        enabled: false