Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deschedule pods that fail to start or restart too often #62

Open
kabakaev opened this issue Dec 6, 2017 · 28 comments
Open

deschedule pods that fail to start or restart too often #62

kabakaev opened this issue Dec 6, 2017 · 28 comments

Comments

@kabakaev
Copy link

@kabakaev kabakaev commented Dec 6, 2017

It is not uncommon that pods get scheduled on nodes that are not able to start it.
For example, a node may have network issues and unable to mount a networked persistent volume, or cannot pull a docker image, or has some docker configuration issue which is seen only on container startup.

Another common issue is when a container gets restarted by liveliness check because of some local node issue (e.g. wrong routing table, slow storage, network latency or packet-drop). In that case, a pod is unhealthy most of the time and hangs in a restart state forever without a chance of being migrated to another node.

As of now, there is no possibility to re-schedule pods with faulty containers. It may be helpful to introduce two new Strategies:

  • container-restart-rate: re-schedule a pod if it is unhealthy since $notReadyPeriod seconds and one of its containers was restarted $maxRestartCount times.
  • pod-startup-failure: a pod was scheduled on a node, but was unable to start all of its containers since $maxStartupTime seconds.

The similar issue is filed against kubernetes: kubernetes/kubernetes#13385

@ravisantoshgudimetla

This comment has been minimized.

Copy link
Contributor

@ravisantoshgudimetla ravisantoshgudimetla commented Jan 3, 2018

Seems like a reasonable ask. @kabakaev I am planning to defer this to 0.6 release or later. Hope you are ok with that.

@ravisantoshgudimetla ravisantoshgudimetla added this to the Future-release milestone Jan 3, 2018
@bgrant0607

This comment has been minimized.

Copy link
Member

@bgrant0607 bgrant0607 commented Jan 22, 2018

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Apr 22, 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented May 22, 2019

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jun 21, 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Jun 21, 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mbdas

This comment has been minimized.

Copy link

@mbdas mbdas commented Sep 9, 2019

A whole bunch of issues were referred to this and then this gets auto closed. Should the users just write a controller and delete pod after too many restarts etc

@mbdas

This comment has been minimized.

Copy link

@mbdas mbdas commented Sep 9, 2019

/reopen

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Sep 9, 2019

@mbdas: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k82cn

This comment has been minimized.

Copy link

@k82cn k82cn commented Sep 10, 2019

/reopen

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Sep 10, 2019

@k82cn: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Sep 10, 2019
@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Oct 10, 2019

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Oct 10, 2019

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ravisantoshgudimetla

This comment has been minimized.

Copy link
Contributor

@ravisantoshgudimetla ravisantoshgudimetla commented Dec 4, 2019

/reopen

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Dec 4, 2019

@ravisantoshgudimetla: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Dec 4, 2019
@ravisantoshgudimetla

This comment has been minimized.

Copy link
Contributor

@ravisantoshgudimetla ravisantoshgudimetla commented Dec 4, 2019

#89 tried addressing this. Let's make sure that we're getting this in before the next release

@fejta-bot

This comment has been minimized.

Copy link

@fejta-bot fejta-bot commented Jan 3, 2020

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Jan 3, 2020

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@damemi

This comment has been minimized.

Copy link
Contributor

@damemi damemi commented Jan 29, 2020

/reopen
/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot reopened this Jan 29, 2020
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

@k8s-ci-robot k8s-ci-robot commented Jan 29, 2020

@damemi: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@seanmalloy

This comment has been minimized.

Copy link
Contributor

@seanmalloy seanmalloy commented Feb 6, 2020

/kind feature

@riking

This comment has been minimized.

Copy link

@riking riking commented Feb 7, 2020

Please remember to implement safeguards against a "pod of death" where attempting to start it crashes kubelet. From what I understand, this would look similar to a pod that simply can't be started on that node.

Wrt restart too often (CrashLoopBackoff), the value of this is questionable because the majority of cases where that happens is an application bug.

@mbdas

This comment has been minimized.

Copy link

@mbdas mbdas commented Feb 7, 2020

True it is app error most times but there are valid scenarios where restarting a sidecar (assuming that crashed) may not fix the pod as a whole and may need a fresh launch and in real life often host misconfigurations/bad upgrade may cause the pod failures and good to move away from those bad hosts in the fleet.

@pohly

This comment has been minimized.

Copy link

@pohly pohly commented Feb 7, 2020

This looks like a reasonable proposal. Ephemeral inline volumes have the same problem: a pod gets scheduled onto a node, then the CSI driver's NodePublishVolume finds that it cannot create the volume, and the pod is stuck.

@pohly

This comment has been minimized.

Copy link

@pohly pohly commented Feb 7, 2020

For my own understanding: is the proposal to delete such a failed pod and then let a higher-level controller (like a stateful set) create a new pod, or is the proposal to just move the pod off the node and schedule it again?

@mbdas

This comment has been minimized.

Copy link

@mbdas mbdas commented Feb 7, 2020

If not mistaken a pod gets scheduled only once for its entire lifetime. So unless its deleted and replaced from a controller/operator, new scheduling will not happen. Now there is a chance the pod maybe scheduled back to the bad node (for that specific use case) but proper fleet management will essentially remove a node that has high rate of failure. But in most cases it will land in a good node. For use cases where a fresh pod launch required any node is fine.

@damemi

This comment has been minimized.

Copy link
Contributor

@damemi damemi commented Feb 7, 2020

It should also be noted that the descheduler only considers pods for eviction that have an ownerreference (unless this is explicitly overridden), so pods that aren't managed by a controller which would attempt to reschedule them would not be evicted by default.

@pohly

This comment has been minimized.

Copy link

@pohly pohly commented Feb 10, 2020

If not mistaken a pod gets scheduled only once for its entire lifetime.

That's what I thought, thanks for the confirmation.

Now there is a chance the pod maybe scheduled back to the bad node (for that specific use case) but proper fleet management will essentially remove a node that has high rate of failure.

That may be true for "broken" nodes, but not for a node that simply doesn't have enough storage capacity left for a certain kind of inline ephemeral volume. I was proposing to add capacity tracking to ensure that Kubernetes will eventually pick a node that has enough capacity, but that KEP has been postponed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.