Update graceful node shutdown docs for beta

[bobbypage]

/sig node

Enhancement Issue - kubernetes/enhancements#2000
Promotion of graceful node shutdown to beta - kubernetes/kubernetes#99735

[k8sio-netlify-preview-bot]

Deploy preview for kubernetes-io-vnext-staging processing.

Building with commit b222a13

https://app.netlify.com/sites/kubernetes-io-vnext-staging/deploys/605d33764045290008b53a37

[reylejano]

/assign @mvortizr

[yvespp]

@bobbypage how do I know if this is working?
I did a shutdown -h now on the node but in the kubelet log I don't see that the kubelet is stopping pods beforehand. I even fails to connect to containerd:

Mar 09 14:06:17 kubelab-worker-e488b83d3ace kubelet[13965]: I0309 14:06:17.737104   13965 operation_generator.go:594] MountVolume.MountDevice succeeded for volume "pvc-...
Mar 09 14:07:14 kubelab-worker-e488b83d3ace kubelet[13965]: W0309 14:07:14.006744   13965 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {/run/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock:>
Mar 09 14:07:14 kubelab-worker-e488b83d3ace kubelet[13965]: W0309 14:07:14.006778   13965 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {/run/containerd/containerd.sock  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial unix /run/containerd/containerd.sock:>
Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
Mar 09 14:07:14 kubelab-worker-e488b83d3ace kubelet[13965]: I0309 14:07:14.010731   13965 dynamic_cafile_content.go:182] Shutting down client-ca-bundle::/etc/kubernetes/pki/ca.crt
Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: kubelet.service: Succeeded.
Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Reboot --
Mar 09 14:08:08 kubelab-worker-e488b83d3ace systemd[1]: Started kubelet: The Kubernetes Node Agent.

Also I can see that the containers get killed by systemd (I have set KillMode=mixed for containerd):

Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: containerd.service: Killing process 10820 (java) with signal SIGKILL.
Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: containerd.service: Killing process 10832 (java) with signal SIGKILL.
Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: containerd.service: Killing process 10835 (java) with signal SIGKILL.
Mar 09 14:07:14 kubelab-worker-e488b83d3ace systemd[1]: containerd.service: Killing process 10900 (java) with signal SIGKILL.

This is with version 1.20.4 with this config:

featureGates:
  GracefulNodeShutdown: true
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s

Also tried setting --feature-gates=GracefulNodeShutdown=true in the kubelet flags.

I can see that the inhibitor is registered:

root@kubelab-worker-e488b83d3ace:~# systemd-inhibit --list
WHO     UID USER PID COMM    WHAT     WHY                                        MODE
kubelet 0   root 945 kubelet shutdown Kubelet needs time to handle node shutdown delay

1 inhibitors listed.

I can open a separate issue if you want to...

[yvespp]

Ok, if I do a systemctl poweroff instead of shutdown -h now it works.
However "Shut Down Guest OS" via VMware talks to the open-vm-tools which in turn executes a shutdown -h now: https://github.com/vmware/open-vm-tools/blob/master/open-vm-tools/lib/system/systemLinux.c#L325
I'll check if shutdown -h now can execute a different systemctl command...

This is on Ubuntu 20.04.2

[bobbypage]

Hi @yvespp

Good to hear you got it working.

systemd inhibitors don't delay the shutdown if you execute shutdown -h now as root xref. systemd will respect the inhibit delay if the shutdown is triggered by a power press, what systemd calls HandlePowerKey (which can be result of an ACPI shutdown event detected by logind from reading /dev/input/...). If you want to test it via cli, I also found doing a scheduled shutdown, i.e. shutdown -h +1, will trigger the inhibit delay.

[yvespp]

@bobbypage VMWare seems to have no way to shut down the vm via power button/ACPI events. It either shuts down the vm via vm-tools or basically just pulls the power.
What worked is replacing /usr/sbin/shutdown whit a custom script and then use "Shut Down Guest OS" via VMware. Script:

#!/bin/bash
exec systemctl poweroff

I looks like there are some features in future versions of systemd that might improve the situation:

• Added option --check-inhibitors for non-tty usage systemd/systemd#18210
• logind: Introduce RebootWithFlags and others systemd/systemd#18316

However I noticed that the pods on the node get deleted but they start immediately again and then get killed by systemd when to os shuts down... This can't be the intention, right?
Non daemon set pods the the status.reason: Shutdown but that doesn't seem to stop Kubernetes to start the pod on the same node again...

[bobbypage]

@yvespp thanks for trying it out. Do you want maybe open a separate github issue so we can debug further and keep this PR focused on the doc changes? Thanks!

[bobbypage]

Updated the PR with update to the docs, this should be ready to review now.

[reylejano]

@kubernetes/sig-node-pr-reviews Can you provide a review?

[sftim]

Some feedback. I'd be willing to merge without seeing that addressed, but I'd definitely prefer to have those changes considered.

[sftim]

I'd still mention that feature gate, but tell readers that it is enabled by default. Not every cluster opts in to every beta feature.

[bobbypage]

Updated with a note regarding feature gate, please take a look.

[sftim]

Please consider moving this much earlier in the page - it looks important!

[bobbypage]

Good point, moved this section earlier.

[sftim]

@bobbypage the PR description mentions that it is a:

Placeholder PR for Graceful Node Shutdown Beta docs

If this is ready for review, I recommend you reword that so it's clear.

[bobbypage]

Thanks @sftim this is ready for review, I updated the PR description and addressed your comments.

[tengqm]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tengqm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• content/en/OWNERS [tengqm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[reylejano]

/lgtm

[reylejano]

/lgtm cancel

[reylejano]

@bobbypage Please rebase

[sftim]

LGTM if rebased the “obvious” way.

[bobbypage]

Rebased, thanks for the review! PTAL again, @sftim @reylejano

[reylejano]

Hi @caesarxuchao @dchen1107 @SergeyKanzhelev @karan @mrunalp @kubernetes/sig-node-pr-reviews , please provide a technical review (tech lgtm) for this PR by March 31 to get this into the release. Thank you!

[sftim]

I'm happy to LGTM this. We can revert if there's been some kind of grave error here!

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 92cf8c32e8807a45156e2ddf470bae55efed9790