Hazelcast 3.5.x Memory Leak using IMap

[jocelynlepage]

We are experimenting memory leaks when using IMaps.

Based on our observations, the leak seems related to the fact that we're regularly destroying maps.
Based on Heap analysis, the culprit seems to be around com.hazelcast.map.impl.MapServiceContextImpl.

Setup:

• Hazelcast 3.5.2, JDK 1.7/1.8, reproduced on Linux/Windows 7 and Mac OSX 10.9.5
• 2 nodes with embedded Hazelcast (e.g. not using separate client JVMs)

Scenario:

• One of our node is creating a new IMap roughly once per minute and put stuff in it at regular interval (few hundreds to few thousands per minute).
• Few seconds/minutes later, the other node is removing data from the map, and if the map is empty, destroys the map.
• We do notice a steady growth of the heap usage, which seems related to Hazelcast IMaps internal management.

We did run a couple of variants with the following observations:

• Hazelcast versions 3.5.1 and 3.5.2 exhibits the same memory leak behavior
• We tested both with JMX/ManCenter enabled and disabled, without changes on the results.
• We disable Map "rotation" (e.g. a single map is used instead) and we didn't noticed the memory leak.
• We also tested with version 3.4.6, problem is NOT present - seems something introduced in 3.5.x

We wanted to update from 3.4.x since as our production system is affected by another memory leak documented here: #4888

Please find a small project on GitHub which demonstrates the problem: https://github.com/jocelynlepage/hz-map-leak
This project includes a couple of heap dump files (produced with jmap) provided for analysis.

Thanks,
Jocelyn

[jerrinot]

@jocelynlepage: Thank you for reporting the issue and for a great reproducer!

I managed to reproduce it. I can see that approximately half of partitions on each member are cleaned, but the other half still retains data. I reckon it's related to backups.

[jerrinot]

Heapdump: http://54.87.52.100/~jara/dumps/issue6317/java_pid14221.0001_.hprof.gz
This is from 3.5.2.

For whoever will fix it: Please ping me so I can delete the dump.

[jerrinot]

I believe it's a side-effect of #5649

Because of this check it MapPartitionDestroyOperation is executed on partition owners only -> The backup replicas are not touched at all. The MapPartitionDestroyOperation should implement the BackupAwareOperation interface.

The question is how to backport it into 3.x branch. As we cannot simply add a new MapPartitionDestroyBackupOperation due compatibility in patch-level releases.

[bwzhang2011]

@jerrinot, any update with such issue ? does it only occur under the specific scenery as the one described above ?

[jerrinot]

@bwzhang2011: PRs fixing the leak in both master and maintenance branches are pending. This fix will be part of Hazelcast 3.5.3 which should be released soon.

[bwzhang2011]

@jerrinot, thanks for feed back and make quick fix and backported plan. looking forward to new hz3.5.3 releasing and more issues to be fixed afterwards.

[jocelynlepage]

Thanks guys for the quick turnaround.
Looking forward to update to 3.5.3 once available.

[IsuraD]

HI jerrinot,

Is this issue fixed in 3.5.3 ?

[jerrinot]

@IsuraD: Hi, it is. See #6400

[dushaniw]

Can I know whether anyone faced a similar issue in 3.0.1 version?

[jerrinot]

@dushaniw 3.10.1 or 3.0.1?

[dushaniw]

Its 3.0.x version.

[fayaz-rvce]

We are facing a similar bug in Production. Even after removing entry from IMap, it can be still seen in heapdump. We are using hazelcast 3.7.2 version? Did anyone face this issue with latest version?