Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spring Boot 2.5.8 graceful shutdown - java.util.concurrent.RejectedExecutionException #956

Closed
michalstefanext opened this issue Feb 18, 2022 · 4 comments
Labels
priority: p2 pubsub type: question Further information is requested

Comments

@michalstefanext
Copy link

I am not 100% sure if this is an issue, but every time I shutdown the spring boot application (version 2.5.8) using:

  • spring.cloud.gcp.pubsub v3.1.0
  • having subscriber using PubSubInboundChannelAdapter (spring integration)
  • using server.shutdown=graceful for spring boot app

I get this exception from Subscriber:

Exception in thread "Thread-35" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3d8416eb[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@705df4f2[Wrapped task = com.google.common.util.concurrent.ListenerCallQueue$PerListenerQueue@5de316dd]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@54d338fa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705)
	at com.google.common.util.concurrent.ListenerCallQueue$PerListenerQueue.dispatch(ListenerCallQueue.java:170)
	at com.google.common.util.concurrent.ListenerCallQueue.dispatch(ListenerCallQueue.java:121)
	at com.google.common.util.concurrent.AbstractService.dispatchListenerEvents(AbstractService.java:516)
	at com.google.common.util.concurrent.AbstractService.notifyFailed(AbstractService.java:479)
	at com.google.api.core.AbstractApiService$InnerService.innerNotifyFailed(AbstractApiService.java:165)
	at com.google.api.core.AbstractApiService$InnerService.access$400(AbstractApiService.java:145)
	at com.google.api.core.AbstractApiService.notifyFailed(AbstractApiService.java:142)
	at com.google.cloud.pubsub.v1.Subscriber.access$1900(Subscriber.java:91)
	at com.google.cloud.pubsub.v1.Subscriber$2.run(Subscriber.java:303)
	at java.base/java.lang.Thread.run(Thread.java:833)

Process finished with exit code 130

Is there a way how to make it shutdown gracefully without exception please?

@elefeint
Copy link
Contributor

RejectedExecutionException means the threadpool is in the process of gracefully shutting down, but was asked to do additional work, and firmly said no.

On the one hand, this is not a problem because it's a normal scenario -- a Pub/Sub message is trying to come in from a gRPC stream. On the other hand, we should probably try to shut down the adapter before the thread pool itself closes to avoid cluttering your logs with exceptions.

@michalstefanext
Copy link
Author

Thanks @elefeint (btw looks like it happens every time, no matter if there's a message coming in from gRPC stream or not).

@meltsufin
Copy link
Member

Thanks for the update! Would you be able to provide some sample code that reproduces this error? Thanks!

@michalstefanext
Copy link
Author

Apologies @meltsufin, we've switched from spring-integration to use spring-cloud-gcp-pubsub directly, so we're no longer facing this issue + we have removed the branch with previous spring-integration code, so I believe it will be better to close this issue, but thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p2 pubsub type: question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants