JvmThreadMetrics has performance issues with many threads

[lpatouchas]

Hello all,
we have an application that by design will have a large number of threads. We use the micrometer library as it is (outside any framework ie, spring). We define a prometheus registry and our prometheus configuration is to poll every 15seconds. We see that when our app exceeds 10-12k threads the application and even tomcat that runs it freezes after startup. If we disable the prometheus polling or remove JvmThreadMetrics the problem does not occure.

The issue is quite easy to reproduce. A simple war that generates 15k running (or even TIMED_WAITING) threads and a curl to /metrics every 5 sec will freeze the app and tomcat immediately after startup (sometimes we might get a response after a while). /metrics, /health and not even http:/hostname:8080 that should be served tomcat respond.

Is this by design, is there a finite amount of threads that this registry supports?

Regards,
Leonidas

[checketts]

That is not by design. Could you provide a sample project that reproduces it?

I suspect it is the jvm.threads.states metric since it iterates through all the threads serially (six times, one for each potential state).

If you could add a MeterFilter that disable the jvm.threads.states gauges let us know if that hang goes away.

[dedousis]

Hello,
In our test case the Tomcat just freezes for a few seconds when we curl it every some seconds.
Below i will attach the sample project we run the tests.
https://github.com/dedousis/JvmThreadMetrics

Regards
Andreas

[mauhiz]

+1. We observed heavy impact at 4K threads, with the metrics pull thread being busy at:

[mauhiz]

This JDK issue sounds related: https://bugs.openjdk.java.net/browse/JDK-8185005

[shakuzen]

Looks like the fix for that JDK issue has been backported to Java 11.0.7. Once that is released, could you see if that makes any difference? With so many threads, it may be best to not use the JvmThreadMetrics. Are you wanting the thread metrics? If not, you can either not bind them or filter them out with a MeterFilter.

[shakuzen]

Has anyone checked with a recent version of the JDK that includes the bug fix mentioned previously? Is the performance okay with that, or is it still an issue with enough threads?

[SoMuchForSubtlety]

Has anyone checked with a recent version of the JDK that includes the bug fix mentioned previously? Is the performance okay with that, or is it still an issue with enough threads?

I'm still seeing this issue with openjdk 11.0.15. jvm.threads is the worst offender, but process.files.open, process.cpu,usage and system.cpu.usage also have a fairly large impact.

Profiling results with ~150 active threads: