Long running instances goes GC-Death because of too many Threads created

[bjoernhaeuser]

Hey,

we are experiencing problems after we upgraded to 0.92.

Eventually the Java-Heap is full and causes Full GC, which makes graylog more or less unuseable. I obtained a Heap-Dump just before I had to restart the application.

The Heap-Dump shows that most of the Heap is spent in the JMX-Beanserver and MetricsRegistry.

After digging into the MetricsRegistry it seems to me that thounsands of threads are created.

We have 1.2 GB Heap and Using G1. This is the graylog2.conf: https://gist.github.com/bjoernhaeuser/90aad72b5dfcdece9016

Currently we have to restart graylog every hours. Prior to that graylog sometimes also stops stream processing and we have to manually restart all the streams by hand.

Thanks
Björn

[kroepke]

This problem is related to our use of InstrumentedThreadExecutor, specifically when used by the stream router and the REST API service.
Both places use cached thread pools to do either time limited calls (for the stream router) or to serve the rest api requests.
However, both use the default pool configuration of core 0, unbounded number of threads and a 60 second idle time, leading to many threads being created of the lifetime of these pools.
See https://gist.github.com/bjoernhaeuser/970b6715891bb7b6c753 for an example.

This in turn leads to individual metrics for each of those threads, which only consume memory and over time it will lead to OOM exceptions and long GC times.

We need to a) limit the number of threads being created and b) prevent too many metrics from being created.
Do we really need to instrument each thread here? For 0.92 my vote would be to remove the instrumentation only, for master I would like to properly configure the pools to not create threads all the time (preferably also fix the number and use a queue for requests). Then instrumenting will not be a problem because the thread names are stable.

Please also review other usages of InstrumentedThreadExecutor, I suspect we have more problems elsewhere that are hard to spot from the threaddumps.

Last but not least: can we somehow check that we don't create endless amounts of metrics? Like regularly checking the number?