Graceful node shutdown

[bobbypage]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubelet can detect and gracefully terminate pods during a node shutdown. Ref: Kubelet should handle node shutdown  kubernetes#91472
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown
• Discussion Link:
• Primary contact (assignee): @bobbypage
• Responsible SIGs: sig/node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (1.20)
  • Beta release target (1.21)
  • Stable release target (TBD)

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[bobbypage]

/sig node

[kikisdeliveryservice]

@bobbypage Do you want this in 1.20 or 1.21?

If so I added a comment here: #2001 (review)

Since you will need to: Update the related Issue to get it in the milestone, add graduation criteria (alpha, beta, etc) and mark this as implementable. by Enhancements Freeze October 6th

[bobbypage]

Hi @kikisdeliveryservice

I'd like to target 1.20 if possible, I've updated the KEP and issue as such, please see the corresponding comment there: #2001 (comment)

Please let me know if anything else is missing.

Thanks for your help!

[kikisdeliveryservice]

Hi @bobbypage !

KEP will need graduation criteria for alpha and in the kep.yaml update to show 1.20 as being alpha.

After that you're good to go =)

[bobbypage]

Thanks @kikisdeliveryservice -- I updated the KEP with that info

[kikisdeliveryservice]

Hi @bobbypage

The KEP seems to be missing graduation criteria for alpha:https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2000-graceful-node-shutdown#graduation-criteria

Please update and merge by October 6th

[kikisdeliveryservice]

That was fast! The KEP is done & good to go for 1.20 👍

[kikisdeliveryservice]

Hi @bobbypage 😄

Since your Enhancement is scheduled to be in 1.20, please keep in mind the important upcoming dates:
Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
Thursday, Nov 12th: Week 9 - Code Freeze

As a reminder, please link all of your k/k PR as well as docs PR to this issue so we can track them.

Thanks!
Kirsten

[annajung]

Hello @bobbypage 👋 , 1.20 Docs lead here.

Does this enhancement work planned for 1.20 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.20 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Nov 6th

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[annajung]

Hi @bobbypage

The docs placeholder deadline is this Friday. Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline

Also, please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for Review

As a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[bobbypage]

Hi @annajung I've created a placeholder PR here: kubernetes/website#24918

The PR for this is still in flight, so I'm not completely sure if it'll make the timeline for 1.20 cut, but will try :). Otherwise, we'll have to target this for 1.21.

[annajung]

Thanks for the update! No worries, you still have a few more days before the code freeze.
The release team will keep track of this until the last day possible, hope you can get it in for 1.20

[kikisdeliveryservice]

Hey @bobbypage

I see that kubernetes/kubernetes#96129 is still open and in active review. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

[kikisdeliveryservice]

Looks like the PR merged! 🥳

[reylejano]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[msau42]

/remove-lifecycle rotten

[rptaylor]

Can there be a way to automatically clean up the pods in shutdown status, perhaps after X hours?
I would almost rather disable graceful node shutdown if it means the cluster gets cluttered with terminated pods that have to be manually cleaned up.

The situation was clarified with kubernetes/website#28235 and https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-garbage-collection, but with the default value of terminated-pod-gc-threshold = 12500, that's an awful lot of dead pods that would linger indefinitely. Also since the cause of the pod termination is clear ( a node shut down), there doesn't seem to be any need to keep the pod indefinitely as there is no need for further diagnostics.

[pacoxu]

Some pending pr or issues related to this KEP:

• Allow extra time for Kubelet to report Pod status when node graceful shutdown kubernetes#110804
• Pods are in Running, Terminating state even after the expiry of shutdownGracePeriod during Graceful Node Shutdown kubernetes#109531
• Pods are getting Scheduled and in Pending state during Graceful Node Shutdown. kubernetes#109450
• Readiness probes should keep running (or pod marked as not ready) during graceful node termination kubernetes#105780
• Graceful node shutdown conflicts with unattended-upgrades package kubernetes#102818

[rhockenbury]

/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[SergeyKanzhelev]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[SergeyKanzhelev]

/remove-lifecycle stale

[SergeyKanzhelev]

What is preventing this from GA-ing? It is entering the perma-beta state, and we try not to do it.

[sftim]

Should we provide Windows support for handling node shutdown gracefully?