OutOfMemory due to messages stored in memory - when no connection to Elastic.

[noban]

OutOfMemory due to messages stored in memory - when no connection to Elastic. Basically we have encountered this situation. Is there any way to configure Stagemonitor not to store these messages, or at least set some limit to count of messages / size of the queue?

[felixbarny]

What do you mean by messages? So your problem was that elasticsearch was unexpectedly down which lead to a OOM Error?

[noban]

We're talking about large amount of FutureTask objects in StageMonitor's ThreadPoolExecutor that appeared in the heapdump. We've had a network problem that caused delivery to fail and as a result workQueue of this executor has consumed all the available memory.

[felixbarny]

I'm sorry for the issues. This should be easy to fix though. I'll probably limit the LinkedBlockingQueue of the Executor to hold max. 1000 Futures.
Thx for the detailed analysis

[noban]

Great to hear, thank you in advance!

[kgusarov]

@felixbarny It would be nice idea to introduce this limit in the configuration.

[felixbarny]

I've just released a new version that contains the fix: https://github.com/stagemonitor/stagemonitor/releases/tag/0.18.0

[noban]

Thank you for a really quick fix!