ehcache3 Client side code Ignoring hash invalidation requests when not fully constructed

[rkavanap]

The following messages are still occassionally seen on ehcache3 clients:

[Clients][client0] [WorkerThread(multi_request_ack_stage, 0)] WARN org.ehcache.clustered.client.internal.store.SimpleClusterTierClientEntity - Ignoring the response org.ehcache.clustered.common.internal.messages.EhcacheEntityResponse$ClientInvalidateHash@1a39c335 as no registered response listener could be found.

This looks like a race condition situation as after this message typically the client timesout awaiting the hash invalidation message.

[rkavanap]

Note that the above was seen in a version that was 1 week old. Testing is in progress on the latest version to see if it still exists.

[rkavanap]

Some more analysis data:

[Clients][client1] org.ehcache.spi.resilience.StoreAccessException: java.util.concurrent.TimeoutException
[Clients][client1] 	at org.ehcache.core.exceptions.StorePassThroughException.handleException(StorePassThroughException.java:78)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.ClusteredStore.putIfAbsent(ClusteredStore.java:268)
[Clients][client1] 	at org.ehcache.core.Ehcache.doPutIfAbsent(Ehcache.java:133)
[Clients][client1] 	at org.ehcache.core.EhcacheBase.putIfAbsent(EhcacheBase.java:296)
[Clients][client1] 	at com.terracottatech.ehcache.clustered.frs.BasicCacheOpsMultiClientIT.runTest(BasicCacheOpsMultiClientIT.java:83)
[Clients][client1] 	at org.terracotta.testing.client.TestClientStub.main(TestClientStub.java:101)
[Clients][client1] Caused by: java.util.concurrent.TimeoutException
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.StrongServerStoreProxy.awaitOnLatch(StrongServerStoreProxy.java:175)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.StrongServerStoreProxy.performWaitingForHashInvalidation(StrongServerStoreProxy.java:114)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.StrongServerStoreProxy.getAndAppend(StrongServerStoreProxy.java:214)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.ReconnectingServerStoreProxy.lambda$getAndAppend$2(ReconnectingServerStoreProxy.java:69)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.ReconnectingServerStoreProxy.onStoreProxy(ReconnectingServerStoreProxy.java:99)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.ReconnectingServerStoreProxy.getAndAppend(ReconnectingServerStoreProxy.java:69)
[Clients][client1] 	at org.ehcache.clustered.client.internal.store.ClusteredStore.putIfAbsent(ClusteredStore.java:251)
[Clients][client1] 	... 4 more

[rkavanap]

From the above set of comments, looks like the following is happening:

Client 1 completed its lifecycle, created its cache and did a putIfAbsent...

In the meantime, client 2 started its lifecycle causing platform to think that another client is connected to this entity. Now this client is not yet ready to accept messages (its response listeners are not setup)..So it totally ignores the ClientInvalidationHash message.

Client 1 times out waiting for the ClientInvalidationDone message

Myself and @AbfrmBlr chatted a bit about this and both of us feel this is where the window exists.

[rkavanap]

just confirmed that this happens with the latest version as well.

[rkavanap]

This issue was not properly addressed as it was later realized that the core may re-order a fetchEntity response with the clientInvalidateHash and after this it was also realized that it is a single threaded SEDA stage, which means a wait inside responselistener is not correct.