Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Graceful node shutdown #2000

Open
bobbypage opened this issue Sep 21, 2020 · 53 comments
Open

Graceful node shutdown #2000

bobbypage opened this issue Sep 21, 2020 · 53 comments
Assignees
Labels
sig/node Categorizes an issue or PR as relevant to SIG Node. stage/beta Denotes an issue tracking an enhancement targeted for Beta status

Comments

@bobbypage
Copy link
Member

bobbypage commented Sep 21, 2020

Enhancement Description

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Sep 21, 2020
@bobbypage
Copy link
Member Author

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Sep 21, 2020
@kikisdeliveryservice kikisdeliveryservice added stage/alpha Denotes an issue tracking an enhancement targeted for Alpha status tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team labels Sep 21, 2020
@kikisdeliveryservice
Copy link
Member

@bobbypage Do you want this in 1.20 or 1.21?

If so I added a comment here: #2001 (review)

Since you will need to: Update the related Issue to get it in the milestone, add graduation criteria (alpha, beta, etc) and mark this as implementable. by Enhancements Freeze October 6th

@bobbypage
Copy link
Member Author

bobbypage commented Oct 2, 2020

Hi @kikisdeliveryservice

I'd like to target 1.20 if possible, I've updated the KEP and issue as such, please see the corresponding comment there: #2001 (comment)

Please let me know if anything else is missing.

Thanks for your help!

@kikisdeliveryservice
Copy link
Member

Hi @bobbypage !

KEP will need graduation criteria for alpha and in the kep.yaml update to show 1.20 as being alpha.

After that you're good to go =)

@kikisdeliveryservice kikisdeliveryservice added tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team and removed tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team labels Oct 2, 2020
@kikisdeliveryservice kikisdeliveryservice added this to the v1.20 milestone Oct 2, 2020
@bobbypage
Copy link
Member Author

bobbypage commented Oct 2, 2020

Thanks @kikisdeliveryservice -- I updated the KEP with that info

@kikisdeliveryservice
Copy link
Member

Hi @bobbypage

The KEP seems to be missing graduation criteria for alpha:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown#graduation-criteria

Please update and merge by October 6th

@kikisdeliveryservice
Copy link
Member

That was fast! The KEP is done & good to go for 1.20 👍

@kikisdeliveryservice
Copy link
Member

Hi @bobbypage 😄

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

@annajung
Copy link
Contributor

Hello @bobbypage 👋 , 1.20 Docs lead here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

@annajung
Copy link
Contributor

annajung commented Nov 1, 2020

Hi @bobbypage

The docs placeholder deadline is this Friday. Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline

Also, please keep in mind the important upcoming dates:

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

@bobbypage
Copy link
Member Author

Hi @annajung I've created a placeholder PR here: kubernetes/website#24918

The PR for this is still in flight, so I'm not completely sure if it'll make the timeline for 1.20 cut, but will try :). Otherwise, we'll have to target this for 1.21.

@annajung
Copy link
Contributor

annajung commented Nov 6, 2020

Thanks for the update! No worries, you still have a few more days before the code freeze.
The release team will keep track of this until the last day possible, hope you can get it in for 1.20

@kikisdeliveryservice
Copy link
Member

Hey @bobbypage

I see that kubernetes/kubernetes#96129 is still open and in active review. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

@kikisdeliveryservice
Copy link
Member

Looks like the PR merged! 🥳

@annajung annajung added tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team and removed tracked/yes Denotes an enhancement issue is actively being tracked by the Release Team labels Jan 7, 2021
@annajung annajung removed this from the v1.20 milestone Jan 7, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 24, 2022
@msau42
Copy link
Member

msau42 commented Jun 24, 2022

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jun 24, 2022
@rptaylor
Copy link

Can there be a way to automatically clean up the pods in shutdown status, perhaps after X hours?
I would almost rather disable graceful node shutdown if it means the cluster gets cluttered with terminated pods that have to be manually cleaned up.

The situation was clarified with kubernetes/website#28235 and https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection, but with the default value of terminated-pod-gc-threshold = 12500, that's an awful lot of dead pods that would linger indefinitely. Also since the cause of the pod termination is clear ( a node shut down), there doesn't seem to be any need to keep the pod indefinitely as there is no need for further diagnostics.

@rhockenbury
Copy link

/milestone clear

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 30, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 29, 2023
@SergeyKanzhelev
Copy link
Member

/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 2, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 3, 2023
@SergeyKanzhelev
Copy link
Member

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 5, 2023
@SergeyKanzhelev
Copy link
Member

What is preventing this from GA-ing? It is entering the perma-beta state, and we try not to do it.

@Atharva-Shinde Atharva-Shinde removed the tracked/no Denotes an enhancement issue is NOT actively being tracked by the Release Team label May 14, 2023
@sftim
Copy link
Contributor

sftim commented Aug 14, 2023

Should we provide Windows support for handling node shutdown gracefully?

@tuibeovince
Copy link
Contributor

@SergeyKanzhelev @mrunalp will this still be a target for GA-ing in v1.30? Thank you

@atiratree
Copy link
Member

There are still outstanding issues that need to be resolved before GA-ing, like kubernetes/kubernetes#122912

@tuibeovince
Copy link
Contributor

I really want to have a clearer picture of the current status of this feature.

As to my knowledge:

the major issues that are currently blocking the GA are

and we have an upcoming Declarative Node Maintenance that will have some overlap with GracefulNodeShutdown
#4212

Is this correct?
Thank you!

@atiratree
Copy link
Member

There are a few more relevant issues, please see the Motivation section of the Declarative Node Maintenance #4213, but I am not sure about the severity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/node Categorizes an issue or PR as relevant to SIG Node. stage/beta Denotes an issue tracking an enhancement targeted for Beta status
Projects
None yet
Development

No branches or pull requests