-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
server awaitTermination() doesn't handle graceful shutdown for open streams #11229
Comments
Could you please try to reproduce this with v1.63.1 or v1.64.0? v1.63.0 contained a few bugs that were fixed in v1.64.0 and backported to v1.63.1: https://github.com/grpc/grpc-java/releases/tag/v1.63.1. |
Thanks for the reply @sergiitk ! |
Adding a shutdown hook that calls shutdown() and await termination() on GRPC server is the correct way to produce a graceful shutdown, as you have already elucidated. We have had discussions in the past on whether to provide this ability in the GRPC server but decided against it since we are a library, not a framework, and we don't control main. |
Thanks, @kannanjgithub. But I'm not sure if you understand the issue from the description. We already invoked |
Added more details @kannanjgithub :
|
We find it surprising that awaitTermination could have stopped working since it works in the examples code. Can you provide a test setup and share the GCP project with us to help debug the issue? |
What version of gRPC-Java are you using?
1.63.0
What is your environment?
RHEL Docker image, JDK 17.
We use https://github.com/grpc-ecosystem/grpc-spring, which uses
awaitTermination
to shut down the server gracefully.What did you expect to see?
gRPC server supports grateful shutdown if we have open streams.
We use gRPC streaming to read and write data via our microservice. We expect that we can utilize K8 graceful shutdown to postpone the pod kill process to finish read/write first and close all streams to don't close the connection.
What did you see instead?
Even if we configured graceful shutdown for gRPC server and K8s pod we still see that gRPC server is terminating immediately after SIGTERM even if we invoke awaitTermination().
Steps to reproduce the bug
You can use
kubectl delete pod
or executekill -TERM PID
for Java process inside your pod (it should have PID 1 if you started your Java app as the main process)See issue: grpc-ecosystem/grpc-spring#1110
See a similar problem described here: https://fedor.medium.com/shutting-down-grpc-services-gracefully-961a95b08f8
The text was updated successfully, but these errors were encountered: