Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Split Brain messages still occupies memory after recovery #10325
When we analyzed our last heap dump (~97% full), we found ~66k messages of
Each message occupied 9kB and most of the message (8KB) consists of
Those messages cannot even be GarbageCollected because of a reference from thread
We also noticed this behavior without server restart. After 26h server uptime there were 18k SplitBrainJoinMessages taking 167MB on heap. No restart was done, only some heavy tests and there are no Hazelcast errors in logs other than "MonitorInvocationsTask/BroadcastOperationControlTask delayed" right before heap dump creation.
What can we do to somehow debug this, or cleanup those messages from heap?
Thanks in advance.
I will try to find out that
and from 66k messages