Max idle timeouts do not remove backups, when EntryProcessor is used to introduce the value

[seckinsq]

We experienced a case on which the backup memory entry was not reducing after the main map entry was removed. The similar problem was mentioned on closed issue #2902 but we have a reproducible scenario when we use EntryProcessor to introduce the value on the map.

Version is 3.10.4 and we use hazelcast-client implementation to connect two member cluster hazelcast.
On member side hazelcast.xml we only defined map with max-idle-seconds extending default map configurations.

<map name="myTestMap">
	<max-idle-seconds>10</max-idle-seconds>
</map>

The entry processor increments an attribute (the first value was given with constructor)

public class MyTestEntryProcessor extends AbstractEntryProcessor<String, Integer> {

	private Integer expectedSeqNr;

	public MyTestEntryProcessor(Integer expectedSeqNr) {
		super();
		this.expectedSeqNr = expectedSeqNr;
	}

	@Override
	public Object process(Entry<String, Integer> entry) {
		if (entry.getValue() == null) {
			entry.setValue(expectedSeqNr);
		} else {
			entry.setValue(entry.getValue() + 1);
		}
		return entry.getValue();
	}
} 

As you can see on below test, we did not use put/set methods but executeOnKey to introduce the value or increment it:

public void testMaxIdleSeconds() throws InterruptedException {
	List<String> keys = new ArrayList<>();
	IMap<String, Integer> myTestMap = HazelcastClient.newHazelcastClient().getMap("myTestMap");
	for (int i = 0; i < 10; i++) {
		String key = RandomStringUtils.randomAlphanumeric(5);
		myTestMap.executeOnKey(key, new MyTestEntryProcessor(i));
		keys.add(key);
	}
	
	for (int i = 0; i < 5; i++) {
		System.out.println("iteration: " + i);
		Thread.sleep(5000);
		for (String key : keys) {
			Integer result = (Integer) myTestMap.executeOnKey(key, new MyTestEntryProcessor(5));
			System.out.println(result);
		}
	}
}

We noticed that the entry numbers and memory is reducing after 10 seconds of idle time. But the backup entry number and memory is being kept.

On same map, if we use put, then both main and backup entry numbers and memory is reduced after max idle seconds. So there is no problem with put. But for our case we wanted to use the lock on key when we used the entry processor.

public void testMaxIdleSecondsWithPut() {
	IMap<String, Integer> myTestMap = HazelcastClient.newHazelcastClient().getMap("myTestMap");
	for (int i = 0; i < 10; i++) {
		myTestMap.put(RandomStringUtils.randomAlphanumeric(5), i);
	}
}

On production environment this caused increase in memory usage after a couple of days and more frequent GC caused a load on CPU and reduced the throughput.
After three days of usage:

[seckinsq]

I forgot to mention, setting TTL fixes the problem. If you set TTL from Hazelcast Management Center then it is effective for new records but old records stay as it was at that moment you set TTL. Currently we plan to resolve this by setting TTL on hazelcast.xml beside max-idle-seconds.

[ahmetmircik]

@seckinsq i ran your test against 3.10.4, 3.11.3 and 3.12. Only with 3.10.4, i replicated the issue and others worked correctly. Using one of the later versions can be an option if it fits your case.

[seckinsq]

Thank you @ahmetmircik , I have just tested with 3.12, indeed it works without problem on this version. I will suggest an upgrade on Hazelcast version, probably we will start testing version 3.12 with our application this week but it may take a couple of weeks to reach on prod. Thanks for your time.

[ahmetmircik]

Good to know that it is working. Closing this issue now.