Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
IAtomicLong corrupt value after memeber leaves cluster #6074
It appears that under some circumstances Hazelcast is getting a corrupt value for IAtomicLongs when a member leaves the cluster.
Hazelcast version is 3.5.1 (I have also verified same behavior in 3.5 and 3.4)
In our actual product we are attempting to use IAtomicLong to keep track of the number of concurrent file transfers to each destination we are sending to. Whenever one of the cluster members shut down we observe several of these counts becoming corrupt. The corrupted values always seem to be negative and generally are between -1 and -10000.
I have created a small test application that is able to produce the same issue.
In this application EdgeAnalog is an analog for the transfer destinations in our real app and FileTransferAnalog is an analog for the file transfers. In this test one thread pushes FileTransferAnalogs to a queue and multiple DistributionJobThreads consume the transfers. Each DistributionJobThread has its own hazlecastInstance. After 30 seconds of steady state I stop one of the DistributionJobThreads and observe several of the IAtomicLongs are corrupt on the next read.
In order to try to isolate the issue as a Hazelcast issue I also update a java.util.concurrent.atomic.AtomicLong in parallel to the Hazelcast IAtomicLong and log when they diverge.