GracefulNodeShutdown feature gate not enabled by default?

[comminutus]

Environmental Info:
K3s Version:

k3s version v1.27.5+k3s1 (8d074ecb)
go version go1.20.7

Node(s) CPU architecture, OS, and Version:

Linux k3s-test-agent-1 6.4.15-200.fc38.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 7 00:25:01 UTC 2023 x86_64 GNU/Linux

Cluster Configuration:
4 servers running in HA, 4 agents, no ServiceLB in favor of kube-vip

Describe the bug:
When shutting down a node, I'd expect the pods scheduled on the node to be rescheduled prior to the node shutting down based on how Graceful Node Shutdown is described. It seems this feature is enabled by default from Kubernetes v1.21 and onward via the GracefulNodeShutdown feature gate.

However, when I shutdown a k3s node, the pods don't get rescheduled at all. They are only terminated and restarted once the node comes back online. If I leave the node offline, the pods still appear to be running from the output of kubectl get pods -A -o wide.

Steps To Reproduce:

1. In one terminal: kubectl get pods -Aw -o wide
2. In another terminal, ssh to one of the nodes running pods and: sudo systemctl shutdown
3. After about a minute, observe the node is in the "Not Ready" state: kubectl get nodes
4. Observe the pods haven't been rescheduled from the first terminal window.
5. Wait about 10 minutes, eventually the pods listed on the node that was shutdown eventually get rescheduled.

Steps to setup cluster:

• Install initial k3s server: curl -sfL https://get.k3s.io | K3S_TOKEN_FILE=my-k3s-token-file.txt INSTALL_K3S_VERSION=v1.27.5+k3s1 sh -s - server --node-taint CriticalAddonsOnly=true:PreferNoSchedule --disable servicelb --tls-san=k3s.local --cluster-init
• Install subsequent servers: curl -sfL https://get.k3s.io | K3S_TOKEN_FILE=my-k3s-token-file.txt INSTALL_K3S_VERSION=v1.27.5+k3s1 sh -s - server --node-taint CriticalAddonsOnly=true:PreferNoSchedule --disable servicelb --tls-san=k3s.local --server https://k3s.local:6443
• Install agents: curl -sfL https://get.k3s.io | K3S_TOKEN_FILE=my-k3s-token-file.txt INSTALL_K3S_VERSION=v1.27.5+k3s1 sh -s - agent --server https://k3s.local:6443

Note: I tried adding --kubelet-arg feature-gates=GracefulNodeShutdown=true to all 3 types of deployments above (initial server, subsequent server, agent) thinking perhaps it's not enabled by default, but this didn't seem to have any effect.

Expected behavior:
From the article mention above, it seems stopping the kubelet service should trigger the pods to be evicted and rescheduled on another node during a grace period before the kubelet service terminates, but that doesn't appear to be the case.

Actual behavior:
The pods running on the node that was shutdown eventually get rescheduled after a long period of not more than 10 minutes.

[brandond]

Graceful Node Shutdown doesn't do anything at all by default, even when enabled. You need to configure it, as described in the docs:

https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/#how-do-i-use-it

[brandond]

Note that by default, both configuration options described above, ShutdownGracePeriod and ShutdownGracePeriodCriticalPods are set to zero, so you will need to configure them as appropriate for your environment to activate graceful node shutdown functionality.

[comminutus]

Thanks @brandond , I was able to setup a kubelet config file (kubelet.yaml) with the following contents:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 90s,
shutdownGracePeriodCriticalPods: 30s

I dropped it into /etc/rancher/k3s. From there I referenced it in the k3s command line installation with --kubelet config=/etc/rancher/k3s/kubelet.yaml and the graceful shutdown worked perfectly when I re-ran the same test.