Hz 3.2.5 stabilizer Final, X-large cluster: OperationTimeoutException: No response for 120000 ms

[Danny-Hazelcast]

Our stabilizer Final profile, XLarge cluster size, test suite, for Hz 3.2.5, Failed
after Aprox: Running 00d 10h 25m

WARN  08:31:02 Failure #1  client:10.63.151.203 Worker exception[com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=TxnLockAndGetOperation{timeout=120000, thread=76}, partitionId=215, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.237.180.39]:5701}, done=false} No response has been send  backups-expected: 0 backups-completed: 0 reinvocations: 0]

Failure[
   message='Worked ran into an unhandled exception'
   type='Worker exception'
   agentAddress=10.63.151.203
   time=Thu Aug 07 08:31:02 UTC 2014
   workerAddress=client:10.63.151.203
   workerId=worker-10.63.151.203-3-client
   test=TestCase{
          id=MapTransactionTest
        , class=com.hazelcast.stabilizer.tests.map.MapTransactionTest
        , basename=MapTransactionTest
        , keyCount=1000
        , threadCount=3
    }
   cause=com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=TxnLockAndGetOper
ation{timeout=120000, thread=76}, partitionId=215, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[10.237.180.39]:5701}, done=false} No response has be
en send  backups-expected: 0 backups-completed: 0 reinvocations: 0
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.newOperationTimeoutException(BasicInvocation.java:782)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:760)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:697)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
    at com.hazelcast.map.tx.TransactionalMapProxySupport.lockAndGet(TransactionalMapProxySupport.java:207)
    at com.hazelcast.map.tx.TransactionalMapProxySupport.getForUpdateInternal(TransactionalMapProxySupport.java:106)
    at com.hazelcast.map.tx.TransactionalMapProxy.getForUpdate(TransactionalMapProxy.java:90)
    at com.hazelcast.map.client.AbstractTxnMapRequest.innerCall(AbstractTxnMapRequest.java:96)
    at com.hazelcast.client.txn.BaseTransactionRequest.call(BaseTransactionRequest.java:40)
    at com.hazelcast.client.CallableClientRequest.process(CallableClientRequest.java:27)
    at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.processRequest(ClientEngineImpl.java:499)
    at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.run(ClientEngineImpl.java:446)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)
    at ------ End remote and begin local stack-trace ------.(Unknown Source)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698)
    at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
    at com.hazelcast.map.tx.TransactionalMapProxySupport.lockAndGet(TransactionalMapProxySupport.java:207)
    at com.hazelcast.map.tx.TransactionalMapProxySupport.getForUpdateInternal(TransactionalMapProxySupport.java:106)
    at com.hazelcast.map.tx.TransactionalMapProxy.getForUpdate(TransactionalMapProxy.java:90)
    at com.hazelcast.map.client.AbstractTxnMapRequest.innerCall(AbstractTxnMapRequest.java:96)
    at com.hazelcast.client.txn.BaseTransactionRequest.call(BaseTransactionRequest.java:40)
    at com.hazelcast.client.CallableClientRequest.process(CallableClientRequest.java:27)
    at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.processRequest(ClientEngineImpl.java:499)
    at com.hazelcast.client.ClientEngineImpl$ClientPacketProcessor.run(ClientEngineImpl.java:446)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.util.executor.PoolExecutorThreadFactory$ManagedThread.run(PoolExecutorThreadFactory.java:59)
    at ------ End remote and begin local stack-trace ------.(Unknown Source)
    at com.hazelcast.client.spi.impl.ClientCallFuture.resolveResponse(ClientCallFuture.java:160)
    at com.hazelcast.client.spi.impl.ClientCallFuture.get(ClientCallFuture.java:109)
    at com.hazelcast.client.spi.impl.ClientCallFuture.get(ClientCallFuture.java:90)
    at com.hazelcast.client.txn.proxy.ClientTxnProxy.invoke(ClientTxnProxy.java:54)
    at com.hazelcast.client.txn.proxy.ClientTxnMapProxy.getForUpdate(ClientTxnMapProxy.java:55)
    at com.hazelcast.stabilizer.tests.map.MapTransactionTest$Worker$1.execute(MapTransactionTest.java:124)
    at com.hazelcast.client.txn.ClientTransactionManager.executeTransaction(ClientTransactionManager.java:80)
    at com.hazelcast.client.txn.ClientTransactionManager.executeTransaction(ClientTransactionManager.java:73)
    at com.hazelcast.client.HazelcastClient.executeTransaction(HazelcastClient.java:299)
    at com.hazelcast.client.HazelcastClientProxy.executeTransaction(HazelcastClientProxy.java:135)
    at com.hazelcast.stabilizer.tests.map.MapTransactionTest$Worker.run(MapTransactionTest.java:120)
    at java.lang.Thread.run(Thread.java:745)
    at com.hazelcast.stabilizer.tests.utils.ThreadSpawner$DefaultThread.run(ThreadSpawner.java:88)

]

Interesting it was the MapTransactionTest, that hit the OperationTimeoutException, in this run and
#3206 run.

could MapTransactionTest be use full in tracking down the OperationTimeoutException ?

other verification AssertionErrors are found, but are not the root cause, and can be ignored
same arguments apply, see #3206

[Danny-Hazelcast]

Full Details
https://s3.amazonaws.com/stabilizer/issues3209/1407362390522.zip

[jerrinot]

In the worker.log files I see "No response for 0 ms".
It should be fixed as of ad975e8

Could you re-run the same test with the latest master?

[jerrinot]

I just noticed this is 3.2.5 test, not master.
The fix has been applied in the maintenance branch in c03584a

[mdogan]

Merged into main operation timeout issue: #2051

[anilgvpcoe]

Hi,

I am using Hazelcast 3.2.6 with 3 nodes for quite sometime now, Today I see that Hazelcast has gone inresponsive and it prints this exception

com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=com.hazelcast.map.operation.MapEntrySetOperation@56f3745e, partitionId=15, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[x.x.x.x]:5706}, done=false} No response has been send backups-expected: 0 backups-completed: 0 reinvocations: 0
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.newOperationTimeoutException(BasicInvocation.java:789) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:753) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:697) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.spi.impl.BasicOperationService.invokeOnPartitions(BasicOperationService.java:623) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.spi.impl.BasicOperationService.invokeOnAllPartitions(BasicOperationService.java:549) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.map.proxy.MapProxySupport.entrySetInternal(MapProxySupport.java:752) ~[hazelcast-3.2.6.jar:3.2.6]
at com.hazelcast.map.proxy.MapProxyImpl.entrySet(MapProxyImpl.java:501) ~[hazelcast-3.2.6.jar:3.2.6]


Can you kindly let me know if the issue was resolved in 3.2.6?