Skip to content

[Graceful Shutdown]Pod is removed from endpoints list for service even "preStop" defined and not finished #67592

Closed
@crowfrog

Description

@crowfrog

Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature

What happened:
Pod is removed from endpoints list for service after user sends command to delete Pod
See:
https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
"Termination of Pods"

  1. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
    1. If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
    2. The processes in the Pod are sent the TERM signal.
  2. (simultaneous with 3) Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.

What you expected to happen:
Pod is removed from endpoints list for service after "preStop" and before send the TERM signal to Pod
hope change to:

  1. (simultaneous with 3) When the Kubelet sees that a Pod has been marked as terminating because the time in 2 has been set, it begins the pod shutdown process.
    1. If the pod has defined a preStop hook, it is invoked inside of the pod. If the preStop hook is still running after the grace period expires, step 2 is then invoked with a small (2 second) extended grace period.
    2. Pod is removed from endpoints list for service, and are no longer considered part of the set of running pods for replication controllers. Pods that shutdown slowly cannot continue to serve traffic as load balancers (like the service proxy) remove them from their rotations.
    3. The processes in the Pod are sent the TERM signal.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
By current terminating process, "preStop" can do limited for graceful shutdown because Pod exit receiving traffic immediately.
I think if we can make such change, the Pod can have more choices to decide exiting behavior by itself.
If Pod did not define "preStop" action, the precess after changed will be same as before.
If Pod define "preStop", the Pod can use "preStop" and "readnessProbe" to decide when Pod stop receiving traffic, when Pod can shutdown gracefully and then kubernetes can send TERM signal to Pod.
This is very useful for developing a strong and graceful application running in a Pod.
Also, current terminating process impact third ingress gateway developing, such as ISTIO.
ISTIO ingress gateway can't check service endpoints list for each request. That will make it distribute some requests to a terminating Pod before ingress gateway got endpoints list update.
That means the traffic will be impacted when we do service "scale in"/"delete Pod" even if we define "preStop" for Pod.

Environment:

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureCategorizes issue or PR as related to a new feature.lifecycle/rottenDenotes an issue or PR that has aged beyond stale and will be auto-closed.sig/networkCategorizes an issue or PR as relevant to SIG Network.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/unresolvedIndicates an issue that can not or will not be resolved.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions