Description
Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature
What happened:
Pod is removed from endpoints list for service after user sends command to delete Pod
See:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
"Termination of Pods"
- (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
- If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
- The processes in the Pod are sent the TERM signal.
- (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
What you expected to happen:
Pod is removed from endpoints list for service after "preStop" and before send the TERM signal to Pod
hope change to:
- (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
- If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
- Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
- The processes in the Pod are sent the TERM signal.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
By current terminating process, "preStop" can do limited for graceful shutdown because Pod exit receiving traffic immediately.
I think if we can make such change, the Pod can have more choices to decide exiting behavior by itself.
If Pod did not define "preStop" action, the precess after changed will be same as before.
If Pod define "preStop", the Pod can use "preStop" and "readnessProbe" to decide when Pod stop receiving traffic, when Pod can shutdown gracefully and then kubernetes can send TERM signal to Pod.
This is very useful for developing a strong and graceful application running in a Pod.
Also, current terminating process impact third ingress gateway developing, such as ISTIO.
ISTIO ingress gateway can't check service endpoints list for each request. That will make it distribute some requests to a terminating Pod before ingress gateway got endpoints list update.
That means the traffic will be impacted when we do service "scale in"/"delete Pod" even if we define "preStop" for Pod.
Environment:
- Kubernetes version (use
kubectl version):
1.11.0 - Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial - Kernel (e.g.
uname -a):
Linux seliius04106 4.4.0-103-generic Suggest people verify they can start a VM on GCE. #126-Ubuntu SMP Mon Dec 4 16:23:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux - Install tools:
- Others: