SEVERE exception when adding a node to a 2 nodes cluster under heavy load

[patriziomunzi]

Hello guys,
we're doing HZ cluster failover tests and we're having very worrying exception when adding a node to a cluster where a node is already receiving high load.
We're using HZ-2.6.8.
Here are the steps we're doing:

1. start node 1
2. start heavy load on node 1
3. start node 2

When the node 2 is started the following exceptions appears in node 1:

23-Apr-2014 15:13:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages  packet=Packet [RESPONSE] name=c:com.hypertv.gaming.model.HTFacebookUserBean, connection=Connection [/46.105.110.104:41190 -> Address[46.105.110.104]:5701] live=true, client=false, type=MEMBER,blockId=112, keySize=153, valueSize=0 client=false obj=null callId=14827297
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
        at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
        at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
        at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
        at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
        at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
        at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
        at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:13:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
        at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
        at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
        at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
        at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
        at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
        at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
        at java.lang.Thread.run(Thread.java:662)

[...]

23-Apr-2014 15:28:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages  packet=Packet [RESPONSE] name=c:com.hypertv.gaming.model.HTFacebookUserBean, connection=Connection [/46.105.110.104:41190 -> Address[46.105.110.104]:5701] live=true, client=false, type=MEMBER,blockId=118, keySize=153, valueSize=0 client=false obj=null callId=16687405
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
        at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
        at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
        at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
        at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
        at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
        at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
        at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:28:11 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
com.hazelcast.core.HazelcastException: CONCURRENT_MAP_PUT_FROM_LOAD handleObjectNoneRedoResponse.responseType 4
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleObjectNoneRedoResponse(BaseManager.java:704)
        at com.hazelcast.impl.BaseManager$ResponseQueueCall.handleNoneRedoResponse(BaseManager.java:716)
        at com.hazelcast.impl.ConcurrentMapManager$MBackupAwareOp.handleNoneRedoResponse(ConcurrentMapManager.java:2241)
        at com.hazelcast.impl.BaseManager$TargetAwareOp.handleResponse(BaseManager.java:819)
        at com.hazelcast.impl.BaseManager.handleResponse(BaseManager.java:1492)
        at com.hazelcast.cluster.ClusterManager$6.process(ClusterManager.java:118)
        at com.hazelcast.cluster.ClusterService.processPacket(ClusterService.java:189)
        at com.hazelcast.cluster.ClusterService.dequeuePackets(ClusterService.java:239)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:204)
        at java.lang.Thread.run(Thread.java:662)

[...]

23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages  processable=com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader@7b1dc9ce
java.lang.NullPointerException
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
        at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
        at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
        at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
        at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] null
java.lang.NullPointerException
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
        at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
        at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
        at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
        at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:43:12 com.hazelcast.impl.ConcurrentMapManager
WARNING: [5.135.142.61]:5701 [htgamingdatagrid] Store thrown exception for CONCURRENT_MAP_GET
java.lang.NullPointerException
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768)
        at com.hazelcast.impl.ConcurrentMapManager.getMap(ConcurrentMapManager.java:2502)
        at com.hazelcast.impl.ConcurrentMapManager.setIndexValues(ConcurrentMapManager.java:1427)
        at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.doMapStoreOperation(ConcurrentMapManager.java:3828)
        at com.hazelcast.impl.ConcurrentMapManager$AbstractMapStoreOperation.run(ConcurrentMapManager.java:3901)
        at com.hazelcast.impl.executor.ParallelExecutorService$ParallelExecutorImpl$ExecutionSegment.run(ParallelExecutorService.java:242)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
        at com.hazelcast.impl.ExecutorThreadFactory$1.run(ExecutorThreadFactory.java:45)
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] error processing messages  processable=com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader@2d7f6fb9
java.lang.NullPointerException
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
        at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
        at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
        at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
        at java.lang.Thread.run(Thread.java:662)
23-Apr-2014 15:43:12 com.hazelcast.cluster.ClusterService
SEVERE: [5.135.142.61]:5701 [htgamingdatagrid] null
java.lang.NullPointerException
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:357)
        at com.hazelcast.impl.BaseManager.returnResponse(BaseManager.java:345)
        at com.hazelcast.impl.ConcurrentMapManager$GetOperationHandler$GetLoader.process(ConcurrentMapManager.java:3838)
        at com.hazelcast.cluster.ClusterService.processProcessable(ClusterService.java:194)
        at com.hazelcast.cluster.ClusterService.dequeueProcessables(ClusterService.java:259)
        at com.hazelcast.cluster.ClusterService.run(ClusterService.java:205)
        at java.lang.Thread.run(Thread.java:662)

[...]

No exceptions in node 2

Any idea???

[enesakar]

Closing the issue. Can you check if issue exists in 3.x versions?