New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Graceful Shutdown]Pod is removed from endpoints list for service even "preStop" defined and not finished #67592
Comments
/sig node |
No any update? |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
I really wish this had some responses! This is exactly the problem we have. There seems to be no way to prevent failures in a pod that is being deleted, it will always continue to receive some traffic during its shutdown phase and because TERM is sent at the same time as the preStop is executed the container just exits. If preStop was called before TERM was sent then this would mean adding sleep into preStop (as described in this blog ) would allow time for the pod to stop processing traffic and therefore not generate errors. |
/reopen I am running a Couchbase cluster and in order to gracefully remove a pod from the cluster I need to run a shutdown script. Any info would be appreciated |
@hatimkapadia2030: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
We're also running into the issue. I think this ticket is worth re-opening. Since we don't have permissions, perhaps we will just re-create? /reopen |
@ottoyiu: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature
What happened:
Pod is removed from endpoints list for service after user sends command to delete Pod
See:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
"Termination of Pods"
What you expected to happen:
Pod is removed from endpoints list for service after "preStop" and before send the TERM signal to Pod
hope change to:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
By current terminating process, "preStop" can do limited for graceful shutdown because Pod exit receiving traffic immediately.
I think if we can make such change, the Pod can have more choices to decide exiting behavior by itself.
If Pod did not define "preStop" action, the precess after changed will be same as before.
If Pod define "preStop", the Pod can use "preStop" and "readnessProbe" to decide when Pod stop receiving traffic, when Pod can shutdown gracefully and then kubernetes can send TERM signal to Pod.
This is very useful for developing a strong and graceful application running in a Pod.
Also, current terminating process impact third ingress gateway developing, such as ISTIO.
ISTIO ingress gateway can't check service endpoints list for each request. That will make it distribute some requests to a terminating Pod before ingress gateway got endpoints list update.
That means the traffic will be impacted when we do service "scale in"/"delete Pod" even if we define "preStop" for Pod.
Environment:
kubectl version
):1.11.0
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
uname -a
):Linux seliius04106 4.4.0-103-generic Suggest people verify they can start a VM on GCE. #126-Ubuntu SMP Mon Dec 4 16:23:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
The text was updated successfully, but these errors were encountered: