Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful node shutdown #2000

Open
bobbypage opened this issue Sep 21, 2020 · 42 comments
Open

Graceful node shutdown #2000

bobbypage opened this issue Sep 21, 2020 · 42 comments
Assignees
Labels
sig/node Categorizes an issue or PR as relevant to SIG Node. stage/beta Denotes an issue tracking an enhancement targeted for Beta status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team

Comments

@bobbypage
Copy link
Member

bobbypage commented Sep 21, 2020

Enhancement Description

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 21, 2020
@bobbypage
Copy link
Member Author

bobbypage commented Sep 21, 2020

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 21, 2020
@kikisdeliveryservice kikisdeliveryservice added stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team labels Sep 21, 2020
@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Oct 1, 2020

@bobbypage Do you want this in 1.20 or 1.21?

If so I added a comment here: #2001 (review)

Since you will need to: Update the related Issue to get it in the milestone, add graduation criteria (alpha, beta, etc) and mark this as implementable. by Enhancements Freeze October 6th

@bobbypage
Copy link
Member Author

bobbypage commented Oct 2, 2020

Hi @kikisdeliveryservice

I'd like to target 1.20 if possible, I've updated the KEP and issue as such, please see the corresponding comment there: #2001 (comment)

Please let me know if anything else is missing.

Thanks for your help!

@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Oct 2, 2020

Hi @bobbypage !

KEP will need graduation criteria for alpha and in the kep.yaml update to show 1.20 as being alpha.

After that you're good to go =)

@kikisdeliveryservice kikisdeliveryservice added tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team and removed tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team labels Oct 2, 2020
@kikisdeliveryservice kikisdeliveryservice added this to the v1.20 milestone Oct 2, 2020
@bobbypage
Copy link
Member Author

bobbypage commented Oct 2, 2020

Thanks @kikisdeliveryservice -- I updated the KEP with that info

@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Oct 2, 2020

Hi @bobbypage

The KEP seems to be missing graduation criteria for alpha:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown#graduation-criteria

Please update and merge by October 6th

@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Oct 2, 2020

That was fast! The KEP is done & good to go for 1.20 👍

@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Oct 11, 2020

Hi @bobbypage 😄

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

@annajung
Copy link
Member

annajung commented Oct 19, 2020

Hello @bobbypage 👋 , 1.20 Docs lead here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

@annajung
Copy link
Member

annajung commented Nov 1, 2020

Hi @bobbypage

The docs placeholder deadline is this Friday. Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline

Also, please keep in mind the important upcoming dates:

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

@bobbypage
Copy link
Member Author

bobbypage commented Nov 6, 2020

Hi @annajung I've created a placeholder PR here: kubernetes/website#24918

The PR for this is still in flight, so I'm not completely sure if it'll make the timeline for 1.20 cut, but will try :). Otherwise, we'll have to target this for 1.21.

@annajung
Copy link
Member

annajung commented Nov 6, 2020

Thanks for the update! No worries, you still have a few more days before the code freeze.
The release team will keep track of this until the last day possible, hope you can get it in for 1.20

@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Nov 10, 2020

Hey @bobbypage

I see that kubernetes/kubernetes#96129 is still open and in active review. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

@kikisdeliveryservice
Copy link
Member

kikisdeliveryservice commented Nov 13, 2020

Looks like the PR merged! 🥳

@annajung annajung added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels Jan 7, 2021
@annajung annajung removed this from the v1.20 milestone Jan 7, 2021
@k8s-triage-robot
Copy link

k8s-triage-robot commented Jul 26, 2021

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 26, 2021
@frittentheke
Copy link

frittentheke commented Jul 31, 2021

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 31, 2021
@SnowmanCharles
Copy link

SnowmanCharles commented Aug 27, 2021

It's nice to see this feature. From this blog, looks like this is already an alpha feature in 1.20 and beta in 1.21.

I'm currently considering to integrate this feature into my cluster. But I did not find any answer for the following question. Could you clarify it here or in the design doc?

Question: Does Graceful Node Shutdown respect Pod Disruption Budget set on pods? If so, how it works? If not, are there any plans to support it?

Reason I'm asking: The users may set Pod Disruption Budget for their work loads. If the user enables Graceful Node Shutdown, they usually will not make efforts to evict pods before shutting down the node. Pod eviction respects Pod Disruption Budget. But if Graceful Node Shutdown does not respect PDB, it means the users' workloads may experience unexpected disruption during node shutdown.

@bobbypage
Copy link
Member Author

bobbypage commented Aug 27, 2021

Hi @SnowmanCharles

Question: Does Graceful Node Shutdown respect Pod Disruption Budget set on pods? If so, how it works? If not, are there any plans to support it?

Graceful Node Shutdown does not respect pod disruption budget. The idea is that the node is already shutting down due to some external factor so there is no way we can respect PDB budget. Rather the goal is give best effort attempt to shutdown pods gracefully. If the goal is safety drain the node and respect PDB, k8s drain / eviction APIs are a better fit.

@SnowmanCharles
Copy link

SnowmanCharles commented Sep 2, 2021

@bobbypage Thanks for details! One more question here. Does this feature finally remove pods(Same as Pod deletion) from kube-apiserver? This is also important because if it does not remove the Pods resource from kube-apiserver, the replicaSet of customers' deployments can not schedule new pods immediately.

@wzshiming
Copy link
Member

wzshiming commented Oct 27, 2021

kubernetes/kubernetes#104798 (comment)

@smarterclayton @bobbypage @mkimuram @SnowmanCharles

I think Kubelet and Scheduler components can treat Shutdown as similar to Evicted.
For Scheduler, the Shutdown Pods are no longer regarded as Ready.
For Kubelet, the Shutdown Pods will no longer be started after the node restarts.

Any ideas?

@k8s-triage-robot
Copy link

k8s-triage-robot commented Jan 25, 2022

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 25, 2022
@k8s-triage-robot
Copy link

k8s-triage-robot commented Feb 24, 2022

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 24, 2022
@reylejano
Copy link
Member

reylejano commented Feb 24, 2022

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 24, 2022
@k8s-triage-robot
Copy link

k8s-triage-robot commented May 25, 2022

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 25, 2022
@k8s-triage-robot
Copy link

k8s-triage-robot commented Jun 24, 2022

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 24, 2022
@msau42
Copy link
Member

msau42 commented Jun 24, 2022

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 24, 2022
@rptaylor
Copy link

rptaylor commented Jul 15, 2022

Can there be a way to automatically clean up the pods in shutdown status, perhaps after X hours?
I would almost rather disable graceful node shutdown if it means the cluster gets cluttered with terminated pods that have to be manually cleaned up.

The situation was clarified with kubernetes/website#28235 and https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection, but with the default value of terminated-pod-gc-threshold = 12500, that's an awful lot of dead pods that would linger indefinitely. Also since the cause of the pod termination is clear ( a node shut down), there doesn't seem to be any need to keep the pod indefinitely as there is no need for further diagnostics.

@rhockenbury
Copy link

rhockenbury commented Oct 1, 2022

/milestone clear

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/node Categorizes an issue or PR as relevant to SIG Node. stage/beta Denotes an issue tracking an enhancement targeted for Beta status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team
Projects
None yet
Development

No branches or pull requests