IMap entry eviction issue

[pramodterdale]

I am using IMap to store huge amount of data and each key has specified time-to-live, but some of the keys are not getting evicted at all even after it passed the time to live value. These keys stays forever, so when i check the map size progarmatically it shows there are entries in the map, same is shown in mancenter but when i try to get the list of keys or values nothing comes back. One more observation is that i have backup enabled with count 1, but for these entries there is no backup as well. I have below properties in hazelcast.xml

<partition-group enabled="true"/>
<map name="default">
  <time-to-live-seconds>0</time-to-live-seconds>
  <eviction-policy>NONE</eviction-policy>
</map>

Code to add messages into the map

     msgMap.put(msgId, value, 30, TimeUnit.SECONDS);

I have registered the EntryListener for eviction and doing the final activity on evicted entries, most of the entries are getting evicted as expected but some entries are never getting evicted. The final processed enrties are less than total input entries and missing count exactly matches with the map entry size at the end. What could be wrong, why are not all the messages are getting evicted. Am i missing somthing? Attaching the mancenter screenshot

   IMap<Long, String> msgMap = getMap("1070810_RM_map");
  System.out.println("MapSize:" + msgMap.size() + " :msgMap:" + msgMap + " :entrySet:" +   msgMap.entrySet());

The output is

    MapSize:433 :msgMap:1070810_EM_map :entrySet:[]

[ahmetmircik]

Hi @pramodterdale, what was your version? Can you test it with latest version 3.7.2?

[pramodterdale]

I am able to reproduce the issue on version 3.7 as well as 3.7.2, below screenshot is from 3.7.2 and last one i provided was from 3.7

[ahmetmircik]

Hi @pramodterdale,

First of all, let me clarify some terms. In hazelcast parlance, expiration and eviction are two different things. Eviction is needed to free memory according to size based policies like map-size or memory-size etc. Expiration is needed to determine life-span of an entry. If your goal is freeing memory you should configure eviction.

In your case, there is no eviction defined since it is NONE --><eviction-policy>NONE</eviction-policy>. But you are using expiration. Hazelcast Map removes expired entries gradually with a background sweeper task. Map size can return those expired but not yet removed entries during that sweep cycle but you can not reach them for ex. via imap#get, eventually all expireds will be removed from map. To accelerate this expiration process please see my comments here and see
this for a detailed explanation about those system properties. (Those system properties are available in version 3.7.2, not included in 3.6)

Btw, i did a test with your given configuration, all entries were swept eventually. Seems no problem.

[pramodterdale]

It does not happen every time and unfortunately i did not see any pattern as well, the volume of data is high about 5k/sec insert, update or delete any of these combinations and the process i ran for couple of hours when i saw this behavior.

I will try the options you have specified in above comments and see if same behavior occurs.

Thanks

[ahmetmircik]

any update?

[pramodterdale]

The above mentioned settings are working fine but still there is a delay, one more thing i noticed is the sweeper task/thread is slow in processing, is there any way to increase number of sweeper tasks/threads, if yes what is the property that i can use..?

[ahmetmircik]

one more thing i noticed is the sweeper task/thread is slow in processing

why do you think it is slow?

[jerrinot]

Hi,

as explained by Ahmet expired entries are not accessible as specified in the expiration interval, however physically they might be removed later. this is working as designed.