Getting No response for 120000 ms and java.util.concurrent.TimeoutException #8621
Labels
Comments
|
Could you tell us more about your config, like
Thx. |
|
No feedback. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After getting a warning of No response for 120000 ms, we are having the exception of Timeout. and one node is going down . Please suggest what can be the issue:
We are running hazelcast on weblogic server.
EDT 2016, timestamp: Thu Jul 28 04:31:35 EDT 2016)
2016-07-28 04:31:36,682 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:37,683 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:38,683 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:39,683 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:40,683 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:40,946 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:31:40 EDT 2016, timestamp: Thu Jul 28 04:31:40 EDT 2016)
2016-07-28 04:31:41,684 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:42,684 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:43,684 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:44,684 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:45,685 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:45,943 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:31:45 EDT 2016, timestamp: Thu Jul 28 04:31:45 EDT 2016)
2016-07-28 04:31:46,685 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:47,685 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:48,686 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:49,686 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:26 timeouts:0 backup-timeouts:0
2016-07-28 04:31:50,686 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:50,945 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:31:50 EDT 2016, timestamp: Thu Jul 28 04:31:50 EDT 2016)
2016-07-28 04:31:51,686 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:52,687 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:53,687 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:54,687 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:55,687 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:55,947 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:31:55 EDT 2016, timestamp: Thu Jul 28 04:31:55 EDT 2016)
2016-07-28 04:31:56,688 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:57,688 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:58,688 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:27 timeouts:0 backup-timeouts:0
2016-07-28 04:31:59,688 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:28 timeouts:0 backup-timeouts:0
2016-07-28 04:32:00,689 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:28 timeouts:0 backup-timeouts:0
2016-07-28 04:32:00,947 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:32:00 EDT 2016, timestamp: Thu Jul 28 04:32:00 EDT 2016)
2016-07-28 04:32:01,689 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:28 timeouts:0 backup-timeouts:0
2016-07-28 04:32:02,689 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:28 timeouts:0 backup-timeouts:0
2016-07-28 04:32:03,689 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:28 timeouts:0 backup-timeouts:0
2016-07-28 04:32:04,690 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:28 timeouts:0 backup-timeouts:0
2016-07-28 04:32:05,690 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:29 timeouts:0 backup-timeouts:0
2016-07-28 04:32:05,947 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:32:05 EDT 2016, timestamp: Thu Jul 28 04:32:05 EDT 2016)
2016-07-28 04:32:06,690 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:29 timeouts:0 backup-timeouts:0
2016-07-28 04:32:07,691 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:29 timeouts:0 backup-timeouts:0
2016-07-28 04:32:08,691 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:29 timeouts:0 backup-timeouts:0
2016-07-28 04:32:09,691 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:29 timeouts:0 backup-timeouts:0
2016-07-28 04:32:10,691 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:29 timeouts:0 backup-timeouts:0
2016-07-28 04:32:10,943 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:32:10 EDT 2016, timestamp: Thu Jul 28 04:32:10 EDT 2016)
2016-07-28 04:32:11,692 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:12,692 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:13,692 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:14,692 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:15,693 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:15,945 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:32:15 EDT 2016, timestamp: Thu Jul 28 04:32:15 EDT 2016)
2016-07-28 04:32:16,693 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:17,693 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:30 timeouts:0 backup-timeouts:0
2016-07-28 04:32:18,693 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:31 timeouts:0 backup-timeouts:0
2016-07-28 04:32:19,694 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:31 timeouts:0 backup-timeouts:0
2016-07-28 04:32:20,694 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:31 timeouts:0 backup-timeouts:0
2016-07-28 04:32:20,947 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:32:20 EDT 2016, timestamp: Thu Jul 28 04:32:20 EDT 2016)
2016-07-28 04:32:21,694 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:31 timeouts:0 backup-timeouts:0
2016-07-28 04:32:22,694 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:31 timeouts:0 backup-timeouts:0
2016-07-28 04:32:23,695 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:32 timeouts:0 backup-timeouts:0
2016-07-28 04:32:24,321 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] No response for 120000 ms. InvocationFuture{invocation=Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.SetOperation{identityHash=932088187, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16561, invocationTime=1469694624317 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}, response=null, done=false}
2016-07-28 04:32:24,322 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Asking if operation execution has been started: Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.SetOperation{identityHash=932088187, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16561, invocationTime=1469694624317 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:24,540 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] No response for 120000 ms. InvocationFuture{invocation=Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1967041492, serviceName='hz:impl:mapService', partitionId=196, replicaIndex=0, callId=16562, invocationTime=1469694624537 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=196, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}, response=null, done=false}
2016-07-28 04:32:24,543 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Asking if operation execution has been started: Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1967041492, serviceName='hz:impl:mapService', partitionId=196, replicaIndex=0, callId=16562, invocationTime=1469694624537 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=196, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:24,543 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] No response for 120000 ms. InvocationFuture{invocation=Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1170306096, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16563, invocationTime=1469694624540 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}, response=null, done=false}
2016-07-28 04:32:24,544 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Asking if operation execution has been started: Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1170306096, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16563, invocationTime=1469694624540 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:24,558 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] No response for 120000 ms. InvocationFuture{invocation=Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=537378939, serviceName='hz:impl:mapService', partitionId=121, replicaIndex=0, callId=16564, invocationTime=1469694624555 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=121, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}, response=null, done=false}
2016-07-28 04:32:24,558 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Asking if operation execution has been started: Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=537378939, serviceName='hz:impl:mapService', partitionId=121, replicaIndex=0, callId=16564, invocationTime=1469694624555 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=121, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:24,695 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:36 timeouts:0 backup-timeouts:0
2016-07-28 04:32:25,695 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:36 timeouts:0 backup-timeouts:0
2016-07-28 04:32:25,945 DEBUG [com.hazelcast.cluster.impl.ClusterHeartbeatManager] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Received heartbeat from Member [nygindapxnp13-o.us.db.com]:12514 (now: Thu Jul 28 04:32:25 EDT 2016, timestamp: Thu Jul 28 04:32:25 EDT 2016)
2016-07-28 04:32:26,282 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] No response for 120000 ms. InvocationFuture{invocation=Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=178256205, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16565, invocationTime=1469694626280 (Thu Jul 28 04:30:26 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}, response=null, done=false}
2016-07-28 04:32:26,283 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Asking if operation execution has been started: Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=178256205, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16565, invocationTime=1469694626280 (Thu Jul 28 04:30:26 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:26,554 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] No response for 120001 ms. InvocationFuture{invocation=Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=869354032, serviceName='hz:impl:mapService', partitionId=103, replicaIndex=0, callId=16566, invocationTime=1469694626551 (Thu Jul 28 04:30:26 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=103, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}, response=null, done=false}
2016-07-28 04:32:26,554 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Asking if operation execution has been started: Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=869354032, serviceName='hz:impl:mapService', partitionId=103, replicaIndex=0, callId=16566, invocationTime=1469694626551 (Thu Jul 28 04:30:26 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=103, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:26,696 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:38 timeouts:0 backup-timeouts:0
2016-07-28 04:32:27,696 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:38 timeouts:0 backup-timeouts:0
2016-07-28 04:32:28,696 DEBUG [com.hazelcast.spi.OperationService] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] Invocations:38 timeouts:0 backup-timeouts:0
2016-07-28 04:32:29,326 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] While asking 'is-executing': Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.SetOperation{identityHash=932088187, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16561, invocationTime=1469694624317 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
java.util.concurrent.TimeoutException: Call Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{identityHash=1744343461, serviceName='hz:impl:mapService', partitionId=-1, replicaIndex=0, callId=16596, invocationTime=1469694744317 (Thu Jul 28 04:32:24 EDT 2016), waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER} encountered a timeout
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:369)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:337)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService.isOperationExecuting(IsStillRunningService.java:85)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:275)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:320)
at com.hazelcast.map.impl.proxy.MapProxySupport.setInternal(MapProxySupport.java:367)
at com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:183)
at com.hazelcast.map.impl.proxy.MapProxyImpl.set(MapProxyImpl.java:173)
at com.hazelcast.spring.cache.HazelcastCache.put(HazelcastCache.java:71)
at org.springframework.cache.interceptor.CacheAspectSupport$CachePutRequest.apply(CacheAspectSupport.java:459)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:226)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:181)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy217.getUserMenu(Unknown Source)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.prepareMenuNodesForTheme(ProvideThemeParametersActionFilter.java:101)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.doFilter(ProvideThemeParametersActionFilter.java:66)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:39)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.util.csrf.filter.CsrfTokenLoadFilter.doFilter(CsrfTokenLoadFilter.java:63)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.BuildInPortletAuthorizationFilter.doFilter(BuildInPortletAuthorizationFilter.java:55)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.TermsAndConditionsFilter.doFilter(TermsAndConditionsFilter.java:56)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.AuthorizationFilter.doFilter(AuthorizationFilter.java:58)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.SSOUserFilter.doFilter(SSOUserFilter.java:73)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.util.csrf.filter.CsrfTokenLoadFilter.doFilter(CsrfTokenLoadFilter.java:63)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3431)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3397)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
2016-07-28 04:32:29,326 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] 'is-executing': false -> Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.SetOperation{identityHash=932088187, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16561, invocationTime=1469694624317 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:29,545 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] While asking 'is-executing': Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1170306096, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16563, invocationTime=1469694624540 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
java.util.concurrent.TimeoutException: Call Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{identityHash=1082653171, serviceName='hz:impl:mapService', partitionId=-1, replicaIndex=0, callId=16598, invocationTime=1469694744540 (Thu Jul 28 04:32:24 EDT 2016), waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER} encountered a timeout
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:369)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:337)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService.isOperationExecuting(IsStillRunningService.java:85)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:275)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:320)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:250)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:94)
at com.hazelcast.spring.cache.HazelcastCache.get(HazelcastCache.java:56)
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:308)
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:279)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:201)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:181)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy235.getPage(Unknown Source)
at com.db.gp.common.util.LocationInfo.getCurrentPage(LocationInfo.java:96)
at com.db.gp.common.util.LocationInfo$$FastClassBySpringCGLIB$$6a12514e.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)
at com.db.gp.common.util.LocationInfo$$EnhancerBySpringCGLIB$$250df9b1.getCurrentPage()
at com.db.gp.portlet.portal.PortletController.index(PortletController.java:60)
at sun.reflect.GeneratedMethodAccessor561.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:852)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:280)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:254)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:136)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:346)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.EnvironmentDetectionFilter.doFilter(EnvironmentDetectionFilter.java:38)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.doFilter(ProvideThemeParametersActionFilter.java:93)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:39)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.util.csrf.filter.CsrfTokenLoadFilter.doFilter(CsrfTokenLoadFilter.java:63)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.BuildInPortletAuthorizationFilter.doFilter(BuildInPortletAuthorizationFilter.java:55)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.TermsAndConditionsFilter.doFilter(TermsAndConditionsFilter.java:56)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.AuthorizationFilter.doFilter(AuthorizationFilter.java:58)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.SSOUserFilter.doFilter(SSOUserFilter.java:73)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.util.csrf.filter.CsrfTokenLoadFilter.doFilter(CsrfTokenLoadFilter.java:63)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3431)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3397)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
2016-07-28 04:32:29,546 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] 'is-executing': false -> Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1170306096, serviceName='hz:impl:mapService', partitionId=251, replicaIndex=0, callId=16563, invocationTime=1469694624540 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=pagesLocation, name=pagesLocation}, partitionId=251, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:29,547 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] While asking 'is-executing': Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1967041492, serviceName='hz:impl:mapService', partitionId=196, replicaIndex=0, callId=16562, invocationTime=1469694624537 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=196, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
java.util.concurrent.TimeoutException: Call Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{identityHash=2041299264, serviceName='hz:impl:mapService', partitionId=-1, replicaIndex=0, callId=16597, invocationTime=1469694744539 (Thu Jul 28 04:32:24 EDT 2016), waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER} encountered a timeout
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:369)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:337)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService.isOperationExecuting(IsStillRunningService.java:85)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:275)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:320)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:250)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:94)
at com.hazelcast.spring.cache.HazelcastCache.get(HazelcastCache.java:56)
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:308)
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:279)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:201)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:181)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy217.getUserMenu(Unknown Source)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.prepareMenuNodesForTheme(ProvideThemeParametersActionFilter.java:101)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.doFilter(ProvideThemeParametersActionFilter.java:66)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.SSOUserFilter.doFilter(SSOUserFilter.java:73)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.util.csrf.filter.CsrfTokenLoadFilter.doFilter(CsrfTokenLoadFilter.java:63)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3431)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3397)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
2016-07-28 04:32:29,547 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] 'is-executing': false -> Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=1967041492, serviceName='hz:impl:mapService', partitionId=196, replicaIndex=0, callId=16562, invocationTime=1469694624537 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=196, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
2016-07-28 04:32:29,560 WARN [com.hazelcast.spi.impl.operationservice.impl.Invocation] - [nygindapxnp11-o.us.db.com]:12514 [dev] [3.6.4] While asking 'is-executing': Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.map.impl.operation.GetOperation{identityHash=537378939, serviceName='hz:impl:mapService', partitionId=121, replicaIndex=0, callId=16564, invocationTime=1469694624555 (Thu Jul 28 04:30:24 EDT 2016), waitTimeout=-1, callTimeout=60000, name=userMenu, name=userMenu}, partitionId=121, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER}
java.util.concurrent.TimeoutException: Call Invocation{serviceName='hz:impl:mapService', op=com.hazelcast.spi.impl.operationservice.impl.operations.IsStillExecutingOperation{identityHash=1829923643, serviceName='hz:impl:mapService', partitionId=-1, replicaIndex=0, callId=16599, invocationTime=1469694744554 (Thu Jul 28 04:32:24 EDT 2016), waitTimeout=-1, callTimeout=5000}, partitionId=-1, replicaIndex=0, tryCount=0, tryPauseMillis=0, invokeCount=1, callTimeout=5000, target=Address[nygindapxnp13-o.us.db.com]:12514, backupsExpected=0, backupsCompleted=0, connection=Connection [/10.151.37.153:12514 -> /10.151.37.154:59783], endpoint=Address[nygindapxnp13-o.us.db.com]:12514, alive=true, type=MEMBER} encountered a timeout
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponse(InvocationFuture.java:369)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveApplicationResponseOrThrowException(InvocationFuture.java:337)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:225)
at com.hazelcast.spi.impl.operationservice.impl.IsStillRunningService.isOperationExecuting(IsStillRunningService.java:85)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.waitForResponse(InvocationFuture.java:275)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:224)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.get(InvocationFuture.java:204)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:320)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:250)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:94)
at com.hazelcast.spring.cache.HazelcastCache.get(HazelcastCache.java:56)
at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:308)
at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:279)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:201)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:181)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
at com.sun.proxy.$Proxy217.getUserMenu(Unknown Source)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.prepareMenuNodesForTheme(ProvideThemeParametersActionFilter.java:101)
at com.db.gp.common.hook.filter.ProvideThemeParametersActionFilter.doFilter(ProvideThemeParametersActionFilter.java:66)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.common.hook.filter.SSOUserFilter.doFilter(SSOUserFilter.java:73)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at com.db.gp.util.csrf.filter.CsrfTokenLoadFilter.doFilter(CsrfTokenLoadFilter.java:63)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:79)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3431)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3397)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2280)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2196)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2174)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1575)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:255)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
The text was updated successfully, but these errors were encountered: