-
Notifications
You must be signed in to change notification settings - Fork 41k
Description
What happened:
setting shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" in /etc/kubernetes/kubelet-config.yaml results in /var/log/kubelet.log messages like:
Jun 2 19:41:48 test-c2rtl5620bsn3hi5voeg-kublettest1-default-0000042f kubelet.service[7402]: E0602 19:41:48.214662 7402 kubelet.go:1404] "Failed to start node shutdown manager" err="node shutdown manager was unable to update logind InhibitDelayMaxSec to 1m0s (ShutdownGracePeriod), current value of InhibitDelayMaxSec (30s) is less than requested ShutdownGracePeriod"
What you expected to happen:
I expected the graceful node shutdown configuration settings to take precedence.
How to reproduce it (as minimally and precisely as possible):
configure graceful node shutdown (shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" ) on a node with the unattended-upgrades package installed
check the log for messages indicating "node shutdown manager was unable to update logind InhibitDelayMaxSec ..."
Anything else we need to know?:
unattended-upgrades writes /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf containing InhibitDelayMaxSec=30
the Kubernetes code writes /etc/systemd/logind.conf.d/99-kubelet.conf which contains a definition of InhibitDelayMaxSec ultimately based on the values in /etc/kubernetes/kubelet-config.yaml
/usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf has a higher precedence than /etc/systemd/logind.conf.d/99-kubelet.conf so the InhibitDelayMaxSec value in /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf is used.
for any value other than 30 seconds ... the Kubernetes implementation of GracefulNodeShutdown is incompatible with the unattended-upgrades package
Environment:
-
Kubernetes version (use
kubectl version
):
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+IKS", GitCommit:"04e16b9d6c7af21e462f7ac675c34b667cd6149e", GitTreeState:"clean", BuildDate:"2021-05-24T08:21:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+IKS", GitCommit:"04e16b9d6c7af21e462f7ac675c34b667cd6149e", GitTreeState:"clean", BuildDate:"2021-05-24T08:19:13Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} -
Cloud provider or hardware configuration:
IBM Cloud -
OS (e.g:
cat /etc/os-release
):
cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.7 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.7 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial -
Kernel (e.g.
uname -a
):
uname -a
Linux test-c2v7ef52068f5kndi8s0-testcluster-default-000002f4.iks.ibm 4.4.0-210-generic Less verbose logging for "normal" things. #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux -
Install tools:
-
Network plugin and version (if this is a network-related bug):
-
Others:
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Activity
kbelow commentedon Jun 11, 2021
/sig sig-node
k8s-ci-robot commentedon Jun 11, 2021
@kbelow: The label(s)
sig/sig-node
cannot be applied, because the repository doesn't have them.In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
kbelow commentedon Jun 11, 2021
/sig node
SergeyKanzhelev commentedon Jun 11, 2021
/cc @bobbypage
KEP: kubernetes/enhancements#2000
pacoxu commentedon Jun 24, 2021
/cc @wzshiming
wzshiming commentedon Jun 24, 2021
/assign
mengjiao-liu commentedon Jun 24, 2021
/triage accepted
45 remaining items