Auto-delete failed Pods

[qiutongs]

Overview

Currently, the only mechanism to auto-delete failed pods is the garbage collection. However, the default threshold is incredibly high - 12500[1]. Therefore it is useless in practice for most of customers.

There are circumstances that pods become failed due to a k8s bug. I have seen two cases so far: "Predicate NodeAffinity failed" pods[2] and OutOfCPU pods[3].

So my question is about if we want to make some components to auto-delete failed pods, like some controllers or even kubelet.

Additional Context

1. lower the default threshold for GC have a saner default value of --terminated-pod-gc-threshold  #78693
2. Kubelet bug leading to failed pods with "Node affinity" error Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet #93338
3. When using kube-proxy as a static pod, there could be a short time that scheduler is not aware of it. Then, scheduler doesn't account for the resource taken by kube-proxy and schedules some pod. But the node rejects the pod because of OutOfCPU.

Updates

04/12/2021

A user has to implement a "watcher" to periodically detect and delete failed pods. That is bad because k8s should be able to take that responsibility.

[alculquicondor]

/sig node
/sig apps

[alculquicondor]

cc @kow3ns @derekwaynecarr

[bobbypage]

/cc

[SergeyKanzhelev]

/cc

[SergeyKanzhelev]

Additional consideration here - have a different thresholds for different failure reasons. In case of pods terminated because of node graceful termination - pods will likely be less interesting for troubleshooting. See #102820

[jinleizh]

/triage accepted
/kind support

[k8s-ci-robot]

@ctrlzhang: The label triage/accepted cannot be applied. Only GitHub organization members can add the label.

In response to this:

/triage accepted
/kind support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wzshiming]

/triage accepted

[ehashman]

/remove-kind support

I don't think this is a support request, I think this is a feature request for making this threshold configurable.

/kind feature

[alculquicondor]

The threshold is actually configurable. The problem is that setting it low potentially makes Jobs unusable.
I think we can only solve this in 1.23 when kubernetes/enhancements#2307 graduates to beta.

[schellj]

Additional consideration here - have a different thresholds for different failure reasons. In case of pods terminated because of node graceful termination - pods will likely be less interesting for troubleshooting. See #102820

Not sure if this is the best place, but we're being affected by this topic and I'd like to to add some context.

GKE users making use of preemptible nodes, which have a lifetime of ~24 hours, are being affected by the change in graceful node shutdown behavior introduced in 1.20.5-gke.500 in that pods scheduled on preemptible nodes that have been shut down do not get deleted. As issue #102820 notes, I was indeed confused by this behavior as the pods and nodes are working as intended, yet the pods are considered "failed". This is also causing such nodes themselves to not get deleted and I've seen new pods get scheduled onto them as well.

[ejose19]

I'm in the same boat as @schellj, besides the kubectl get pod clutter, it also stop other commands from working without any extra steps (like kubectl logs) so one needs to manually remove all shutdown pods or register a job for it. A configurable threshold for just shutdown pods (like @SergeyKanzhelev mentioned) seems like a must (and probably, most users would expect that shutdown pods gets auto-deleted by default).

[alculquicondor]

Note that if we deliver a solution just for shutdown pods in kubernetes, it would arrive at 1.24 at the earliest (it's a new feature and needs to go through the KEP process). By then, the Job API will be fixed and we can already lower the threshold.

[schellj]

@alculquicondor Understood. At least in my mind, having a lower gc threshold doesn't entirely solve the issue with shutdown pods as those are pods that have behaved as intended and shouldn't be considered failed and kept around in the first place.

[alculquicondor]

cc @bobbypage to answer why the pods are not simply deleted.

[SergeyKanzhelev]

those are pods that have behaved as intended and shouldn't be considered failed

this statement is questionable. We just don't know in some cases. If it was a job that hasn't finished yet, it is failed, even with the clean termination. If the pod failed to clearly terminate, we also want to keep information about that pod around. It is clear that pods that were terminated with the graceful termination are "happier" then ones that crashed on their own. But it is clearly not 100% expected behavior in a general sense.

That's said, better cleanup rules might be beneficial here for sure.

[alculquicondor]

If it was a job that hasn't finished yet, it is failed, even with the clean termination.

The fixed Job controller will consider any pod deletion a failure, even after the pod is completely removed from the API.

[bobbypage]

Thanks @schellj for the feedback. We are looking into the behavior for handling pods on shutdown.

In the original design of the graceful node shutdown KEP it was decided to not explicitly delete pods, but rather put pods into failed phase. This followed the pattern for kubelet evictions and also made it possible for users to see why their pods were terminated. If they were deleted, pods could appear to vanish suddenly without explanation why which could also appear to be confusing. We are seeing if this makes sense as behavior long term.

preferable for the pods to simply be deleted, as they were in versions of kubernetes prior to 1.20.5.

I'm not super clear what you're referring to here... what mechanism are you suggesting was used to delete pods prior to 1.20.5?

[schellj]

@bobbypage I'm not entirely sure, but I'm guessing that on GKE prior to 1.20.5-gke.500, there was a different process that would cordon and drain their preemptible nodes.

[jameskunc]

@bobbypage

This followed the pattern for kubelet evictions and also made it possible for users to see why their pods were terminated. If they were deleted, pods could appear to vanish suddenly without explanation why which could also appear to be confusing.

Having a terminated-pod-gc-expiration configuration option in conjunction with the terminated-pod-gc-threshold would work for us, then we can still debug the failed ones but they go away after some time.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[schellj]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[schellj]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[schellj]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[pcj]

/reopen

[k8s-ci-robot]

@pcj: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zorgzerg]

/reopen

[k8s-ci-robot]

@zorgzerg: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.