[Graceful Shutdown]Pod is removed from endpoints list for service even "preStop" defined and not finished

[crowfrog]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature

What happened:
Pod is removed from endpoints list for service after user sends command to delete Pod
See:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
"Termination of Pods"

4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
   1. If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
   2. The processes in the Pod are sent the TERM signal.
5. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.

What you expected to happen:
Pod is removed from endpoints list for service after "preStop" and before send the TERM signal to Pod
hope change to:

4. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
   1. If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
   2. Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
   3. The processes in the Pod are sent the TERM signal.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
By current terminating process, "preStop" can do limited for graceful shutdown because Pod exit receiving traffic immediately.
I think if we can make such change, the Pod can have more choices to decide exiting behavior by itself.
If Pod did not define "preStop" action, the precess after changed will be same as before.
If Pod define "preStop", the Pod can use "preStop" and "readnessProbe" to decide when Pod stop receiving traffic, when Pod can shutdown gracefully and then kubernetes can send TERM signal to Pod.
This is very useful for developing a strong and graceful application running in a Pod.
Also, current terminating process impact third ingress gateway developing, such as ISTIO.
ISTIO ingress gateway can't check service endpoints list for each request. That will make it distribute some requests to a terminating Pod before ingress gateway got endpoints list update.
That means the traffic will be impacted when we do service "scale in"/"delete Pod" even if we define "preStop" for Pod.

Environment:

• Kubernetes version (use kubectl version):
  1.11.0
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
  NAME="Ubuntu"
  VERSION="16.04.3 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.3 LTS"
  VERSION_ID="16.04"
  HOME_URL="http://www.ubuntu.com/"
  SUPPORT_URL="http://help.ubuntu.com/"
  BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
  VERSION_CODENAME=xenial
  UBUNTU_CODENAME=xenial
• Kernel (e.g. uname -a):
  Linux seliius04106 4.4.0-103-generic Suggest people verify they can start a VM on GCE. #126-Ubuntu SMP Mon Dec 4 16:23:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

[neolit123]

/sig node

[crowfrog]

No any update?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MartinVUALTO]

I really wish this had some responses! This is exactly the problem we have. There seems to be no way to prevent failures in a pod that is being deleted, it will always continue to receive some traffic during its shutdown phase and because TERM is sent at the same time as the preStop is executed the container just exits. If preStop was called before TERM was sent then this would mean adding sleep into preStop (as described in this blog ) would allow time for the pod to stop processing traffic and therefore not generate errors.
This is currently a serious problem for us as we use HPA and when the load starts to drop we will always get a flurry of failures because the load is spread evenly between all the pods and when HPA removes the unnecessary pods it will always generate some fails.

[hatimkapadia2030]

/reopen
Hello, Is there any update on this. We are having a similar issue.

I am running a Couchbase cluster and in order to gracefully remove a pod from the cluster I need to run a shutdown script.
Since the scripts depends on resolving the IP using the endpoints the script always fails and there is no graceful shutdown.

Any info would be appreciated

[k8s-ci-robot]

@hatimkapadia2030: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
Hello, Is there any update on this. We are having a similar issue.

I am running a Couchbase cluster and in order to gracefully remove a pod from the cluster I need to run a shutdown script.
Since the scripts depends on resolving the IP using the endpoints the script always fails and there is no graceful shutdown.

Any info would be appreciated

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ottoyiu]

We're also running into the issue. I think this ticket is worth re-opening. Since we don't have permissions, perhaps we will just re-create?

/reopen

[k8s-ci-robot]

@ottoyiu: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

We're also running into the issue. I think this ticket is worth re-opening. Since we don't have permissions, perhaps we will just re-create?

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.