Graceful node shutdown conflicts with unattended-upgrades package

[kbelow]

What happened:

setting shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" in /etc/kubernetes/kubelet-config.yaml results in /var/log/kubelet.log messages like:
Jun 2 19:41:48 test-c2rtl5620bsn3hi5voeg-kublettest1-default-0000042f kubelet.service[7402]: E0602 19:41:48.214662 7402 kubelet.go:1404] "Failed to start node shutdown manager" err="node shutdown manager was unable to update logind InhibitDelayMaxSec to 1m0s (ShutdownGracePeriod), current value of InhibitDelayMaxSec (30s) is less than requested ShutdownGracePeriod"

What you expected to happen:

I expected the graceful node shutdown configuration settings to take precedence.

How to reproduce it (as minimally and precisely as possible):

configure graceful node shutdown (shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" ) on a node with the unattended-upgrades package installed

check the log for messages indicating "node shutdown manager was unable to update logind InhibitDelayMaxSec ..."

Anything else we need to know?:

unattended-upgrades writes /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf containing InhibitDelayMaxSec=30

the Kubernetes code writes /etc/systemd/logind.conf.d/99-kubelet.conf which contains a definition of InhibitDelayMaxSec ultimately based on the values in /etc/kubernetes/kubelet-config.yaml

/usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf has a higher precedence than /etc/systemd/logind.conf.d/99-kubelet.conf so the InhibitDelayMaxSec value in /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf is used.

for any value other than 30 seconds ... the Kubernetes implementation of GracefulNodeShutdown is incompatible with the unattended-upgrades package

Environment:

• Kubernetes version (use kubectl version):
  kubectl version
  Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+IKS", GitCommit:"04e16b9d6c7af21e462f7ac675c34b667cd6149e", GitTreeState:"clean", BuildDate:"2021-05-24T08:21:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+IKS", GitCommit:"04e16b9d6c7af21e462f7ac675c34b667cd6149e", GitTreeState:"clean", BuildDate:"2021-05-24T08:19:13Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  IBM Cloud

• OS (e.g: cat /etc/os-release):
  cat /etc/os-release
  NAME="Ubuntu"
  VERSION="16.04.7 LTS (Xenial Xerus)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 16.04.7 LTS"
  VERSION_ID="16.04"
  HOME_URL="http://www.ubuntu.com/"
  SUPPORT_URL="http://help.ubuntu.com/"
  BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
  VERSION_CODENAME=xenial
  UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):
  uname -a
  Linux test-c2v7ef52068f5kndi8s0-testcluster-default-000002f4.iks.ibm 4.4.0-210-generic Less verbose logging for "normal" things. #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Network plugin and version (if this is a network-related bug):

• Others:

[kbelow]

/sig sig-node

[k8s-ci-robot]

@kbelow: The label(s) sig/sig-node cannot be applied, because the repository doesn't have them.

In response to this:

/sig sig-node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kbelow]

/sig node

[SergeyKanzhelev]

/cc @bobbypage

KEP: kubernetes/enhancements#2000

[pacoxu]

/cc @wzshiming

[wzshiming]

/assign

[mengjiao-liu]

/triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[bobbypage]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vaibhav2107]

/remove-lifecycle stale

[kbelow]

I had an opportunity to test shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" with the change made in PR #103137

The outcome has not changed since this issue was created ...

when configuring graceful node shutdown (shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" ) on a node with the unattended-upgrades package installed ...

the Kubernetes code writes /etc/systemd/logind.conf.d/99-kubelet.conf which contains a definition of InhibitDelayMaxSec ultimately based on the values in /etc/kubernetes/kubelet-config.yaml

/usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf has a higher precedence than /etc/systemd/logind.conf.d/99-kubelet.conf so the InhibitDelayMaxSec value in /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf is used.

An error is logged in /var/log/kubelet.log:
Feb  3 20:45:14 test-c7u14t620vqge83-test2497-default-0000033a kubelet.service[117070]: I0203 20:45:14.748748  117070 nodeshutdown_manager_linux.go:124] "Creating node shutdown manager" shutdownGracePeriodRequested="1m0s" shutdownGracePeriodCriticalPods="30s" shutdownGracePeriodByPodPriority=[{Priority:0 ShutdownGracePeriodSeconds:30} {Priority:2000000000 ShutdownGracePeriodSeconds:30}]
Feb  3 20:45:15 test-c7u14t620vqge83-test2497-default-0000033a kubelet.service[117070]: E0203 20:45:15.085043  117070 kubelet.go:1451] "Failed to start node shutdown manager" err="node shutdown manager was unable to update logind InhibitDelayMaxSec to 1m0s (ShutdownGracePeriod), current value of InhibitDelayMaxSec (30s) is less than requested ShutdownGracePeriod"

After more carefully reading #103137, I would agree with a comment from bobby page

 commented [on Jun 28, 2021](https://github.com/kubernetes/kubernetes/issues/102818#issuecomment-870149187)  
This change makes sense, but I don't think it will fix https://github.com/kubernetes/kubernetes/issues/102818

For reference ... this page has a good description of configuration precedence https://www.freedesktop.org/software/systemd/man/logind.conf.html

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rtheis]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rtheis]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rtheis]

/remove-lifecycle stale

[kbelow]

I had an opportunity to test shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" with a newer version ... the problem continues to exist.

Environment:

Kubernetes version (use kubectl version):
kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0+IKS", GitCommit:"1975a162518c7d93fe4369be290092a0d7478623", GitTreeState:"clean", BuildDate:"2022-12-09T12:45:07Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0+IKS", GitCommit:"1975a162518c7d93fe4369be290092a0d7478623", GitTreeState:"clean", BuildDate:"2022-12-09T12:38:38Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration:
IBM Cloud

OS (e.g: cat /etc/os-release):
cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Kernel (e.g. uname -a):
uname -a
Linux test-ceuthic20s1c1ehbqe90-graceshutdo-default-00000212 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[tuibeovince]

the Kubernetes code writes /etc/systemd/logind.conf.d/99-kubelet.conf which contains a definition of InhibitDelayMaxSec ultimately based on the values in /etc/kubernetes/kubelet-config.yaml

/usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf has a higher precedence than /etc/systemd/logind.conf.d/99-kubelet.conf so the InhibitDelayMaxSec value in /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf is used.

If I am understanding the implementation correctly, lexographically the 99-kubelet.conf filename puts the precedence of unattended-upgrades-logind-maxdelay.conf dropin to be higher. I think a change of the prefix of the kubelet drop in settings would give it better precedence, but the scope of effect of such precedence change would need to be looked into.

for any value other than 30 seconds ... the Kubernetes implementation of GracefulNodeShutdown is incompatible with the unattended-upgrades package

I would like to clarify the intention of this sentence. Does it mean that since unattended-upgrades package is setting InhibitDelayMaxSec to 30s, GracefulNodeShutdown will fail if it is not set to 30s.

[tuibeovince]

Just tested this on my dev environment;
the issue still exists with Ubuntu 22.04.2

# kubectl version

Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.8

# cat /etc/os-release

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy