Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OverloadException on shutdownCluster #13730

Closed
Danny-Hazelcast opened this issue Sep 11, 2018 · 1 comment
Closed

OverloadException on shutdownCluster #13730

Danny-Hazelcast opened this issue Sep 11, 2018 · 1 comment

Comments

@Danny-Hazelcast
Copy link
Member

@Danny-Hazelcast Danny-Hazelcast commented Sep 11, 2018

trying to shutdown my cluster but hazelcast cluster gives HazelcastOverloadException,
and will not do a graceful shutdown when requested.

calling cluster shutdown is sync call and could be considered 'high priority',

https://hazelcast-l337.ci.cloudbees.com/view/stable/job/stable-cleanShutdown/2/console

/disk1/jenkins/workspace/stable-cleanShutdown/3.11-SNAPSHOT/2018_09_10-16_15_03/stable/async/async-bak/put-hd/big-val

fail HzMember1HZ clusterShutdown hzcmd.ops.ClusterShutDown threadId=0 com.hazelcast.core.HazelcastOverloadException: Failed to start invocation due to overload: Invocation{op=com.hazelcast.internal.cluster.impl.operations.LockClusterStateOp{serviceName='hz:core:clusterService', identityHash=682232950, partitionId=-1, replicaIndex=0, callId=0, invocationTime=1536586084033 (2018-09-10 13:28:04.033), waitTimeout=-1, callTimeout=60000}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1536586084031, firstInvocationTime='2018-09-10 13:28:04.031', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[10.0.0.87]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}

http://54.82.84.143/~jenkins/workspace/stable-cleanShutdown/3.11-SNAPSHOT/2018_09_10-16_15_03/stable/async/async-bak/put-hd/big-val

cat ./output/HZ/HzMember1HZ/exception.txt
com.hazelcast.core.HazelcastOverloadException: Failed to start invocation due to overload: Invocation{op=com.hazelcast.internal.cluster.impl.operations.LockClusterStateOp{serviceName='hz:core:clusterService', identityHash=682232950, partitionId=-1, replicaIndex=0, callId=0, invocationTime=1536586084033 (2018-09-10 13:28:04.033), waitTimeout=-1, callTimeout=60000}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1536586084031, firstInvocationTime='2018-09-10 13:28:04.031', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[10.0.0.87]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}
	at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:117)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:559)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:530)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:220)
	at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnTarget(OperationServiceImpl.java:327)
	at com.hazelcast.internal.cluster.impl.ClusterStateManager.lockClusterStateOnAllMembers(ClusterStateManager.java:413)
	at com.hazelcast.internal.cluster.impl.ClusterStateManager.changeClusterState(ClusterStateManager.java:367)
	at com.hazelcast.internal.cluster.impl.ClusterStateManager.changeClusterState(ClusterStateManager.java:345)
	at com.hazelcast.internal.cluster.impl.ClusterServiceImpl.changeClusterState(ClusterServiceImpl.java:854)
	at com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdownCluster(ClusterServiceImpl.java:933)
	at com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdown(ClusterServiceImpl.java:923)
	at hzcmd.ops.ClusterShutDown.timeStep(ClusterShutDown.java:8)
	at remote.bench.marker.MetricsMarker.flatOut(MetricsMarker.java:53)
	at remote.bench.marker.MetricsMarker.bench(MetricsMarker.java:40)
	at remote.bench.BenchThread.call(BenchThread.java:42)
	at remote.bench.BenchThread.call(BenchThread.java:15)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.hazelcast.core.HazelcastOverloadException: Timed out trying to acquire another call ID. maxConcurrentInvocations = 9, backoffTimeout = 60000 msecs, elapsed:60169 msecs
	at com.hazelcast.spi.impl.sequence.CallIdSequenceWithBackpressure.handleNoSpaceLeft(CallIdSequenceWithBackpressure.java:61)
	at com.hazelcast.spi.impl.sequence.AbstractCallIdSequence.next(AbstractCallIdSequence.java:56)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:115)
	... 21 more
com.hazelcast.core.HazelcastOverloadException: Failed to start invocation due to overload: Invocation{op=com.hazelcast.internal.cluster.impl.operations.LockClusterStateOp{serviceName='hz:core:clusterService', identityHash=682232950, partitionId=-1, replicaIndex=0, callId=0, invocationTime=1536586084033 (2018-09-10 13:28:04.033), waitTimeout=-1, callTimeout=60000}, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeoutMillis=60000, firstInvocationTimeMs=1536586084031, firstInvocationTime='2018-09-10 13:28:04.031', lastHeartbeatMillis=0, lastHeartbeatTime='1970-01-01 00:00:00.000', target=[10.0.0.87]:5701, pendingResponse={VOID}, backupsAcksExpected=0, backupsAcksReceived=0, connection=null}
	at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:117)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:559)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:530)
	at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:220)
	at com.hazelcast.spi.impl.operationservice.impl.OperationServiceImpl.invokeOnTarget(OperationServiceImpl.java:327)
	at com.hazelcast.internal.cluster.impl.ClusterStateManager.lockClusterStateOnAllMembers(ClusterStateManager.java:413)
	at com.hazelcast.internal.cluster.impl.ClusterStateManager.changeClusterState(ClusterStateManager.java:367)
	at com.hazelcast.internal.cluster.impl.ClusterStateManager.changeClusterState(ClusterStateManager.java:345)
	at com.hazelcast.internal.cluster.impl.ClusterServiceImpl.changeClusterState(ClusterServiceImpl.java:854)
	at com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdownCluster(ClusterServiceImpl.java:933)
	at com.hazelcast.internal.cluster.impl.ClusterServiceImpl.shutdown(ClusterServiceImpl.java:923)
	at hzcmd.ops.ClusterShutDown.timeStep(ClusterShutDown.java:8)
	at remote.bench.marker.MetricsMarker.flatOut(MetricsMarker.java:53)
	at remote.bench.marker.MetricsMarker.bench(MetricsMarker.java:40)
	at remote.bench.BenchThread.call(BenchThread.java:42)
	at remote.bench.BenchThread.call(BenchThread.java:15)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.hazelcast.core.HazelcastOverloadException: Timed out trying to acquire another call ID. maxConcurrentInvocations = 9, backoffTimeout = 60000 msecs, elapsed:60169 msecs
	at com.hazelcast.spi.impl.sequence.CallIdSequenceWithBackpressure.handleNoSpaceLeft(CallIdSequenceWithBackpressure.java:61)
	at com.hazelcast.spi.impl.sequence.AbstractCallIdSequence.next(AbstractCallIdSequence.java:56)
	at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:115)
	... 21 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.