javaagent causes big heap on high load for Netty instrumentation

[ozoz03]

Javaagent produces a critical heap increase for Netty instrumentation on our prod env (heap grows persistently over days and is not cleaned up by GC).
We was able to reproduce the issue by a test app with artillery load.

Looks like cache that contains FutureListenerWrappers is not properly maintained by GC:

Steps to reproduce
Here are steps to reproduce: https://github.com/ozoz03/netty-otel-memory-leak/blob/master/README.md

What version are you using?
v1.10.0

Environment
Compiler: openjdk version "11.0.11" 2021-04-20 LTS
OS: macOS 11.6.2, Alpine Linux 3.12.8

[laurit]

@ozoz03 I ran your sample app and am not convinced that this is a memory leak. If I run your sample app for a bit and perform gc from jconsole before taking a heap dump then all the FutureListenerWrappers are gone. Have you observed an OutOfMemoryError? Or are you just concerned about the space consumed by FutureListenerWrappers.

[ozoz03]

@laurit on prod we cannot wait until the oom error happened. we just redeploy.
the problem as I see it, is that gc does not run a full collection by itself and allows heap to grow uncontrolled.

[iNikem]

@ozoz03 How large heap do you use in your production?

[ozoz03]

@iNikem 6g

[laurit]

@ozoz03 If possible could you verify whether the issue you observed is resolved in nightly build https://oss.sonatype.org/content/repositories/snapshots/io/opentelemetry/javaagent/opentelemetry-javaagent/1.11.0-SNAPSHOT/

[ozoz03]

@laurit thank you a lot.
going to test the build.
will be back in 2-3 days.

[ozoz03]

@laurit hey, sorry for the delay.

from what we could see in our graphs the memory leak was still in place with the new snapshot version.
but we believe we found the root cause for the issue and proposed the fix in this PR