Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
GitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
MulticastJoiner may cause OOME because of the self-amplification in SplitBrainJoinMessage sending #11836
A user reported the issue below:
I have a 2 members Hazelcast cluster on the same computer.
There a few computers (about 5-6 other developers' machines) in the same network with similar clusters (but different ones, that is they have different group name/password).
The seems to lead to a situation when
I can see in my log, thousands of similar messages:
Eventually, free heap runs out and CPU utilization goes to 100% and "OutOfMemoryError: GC overhead limit exceeded" ensue.
In my opinion, the problem might be in sending of another
Here is my assumption:
Provided some clusters are running for a long time, I think this is what I see when I start the hazelcast member and see that it is flooded with messages and OOMEs in virtually minutes.
I tried experimenting with a small test program at home and while I don't see the same magnitudes, I see the increase in the amount of messages accumulating in the deque with each memory snapshot I make in one of the clusters. Probably, give it time and it will eventually result in the same situation.