Skip to content

Graceful node shutdown conflicts with unattended-upgrades package #102818

@kbelow

Description

@kbelow

What happened:

setting shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" in /etc/kubernetes/kubelet-config.yaml results in /var/log/kubelet.log messages like:
Jun 2 19:41:48 test-c2rtl5620bsn3hi5voeg-kublettest1-default-0000042f kubelet.service[7402]: E0602 19:41:48.214662 7402 kubelet.go:1404] "Failed to start node shutdown manager" err="node shutdown manager was unable to update logind InhibitDelayMaxSec to 1m0s (ShutdownGracePeriod), current value of InhibitDelayMaxSec (30s) is less than requested ShutdownGracePeriod"

What you expected to happen:

I expected the graceful node shutdown configuration settings to take precedence.

How to reproduce it (as minimally and precisely as possible):

configure graceful node shutdown (shutdownGracePeriod: "60s" and shutdownGracePeriodCriticalPods: "30s" ) on a node with the unattended-upgrades package installed

check the log for messages indicating "node shutdown manager was unable to update logind InhibitDelayMaxSec ..."

Anything else we need to know?:

unattended-upgrades writes /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf containing InhibitDelayMaxSec=30

the Kubernetes code writes /etc/systemd/logind.conf.d/99-kubelet.conf which contains a definition of InhibitDelayMaxSec ultimately based on the values in /etc/kubernetes/kubelet-config.yaml

/usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf has a higher precedence than /etc/systemd/logind.conf.d/99-kubelet.conf so the InhibitDelayMaxSec value in /usr/lib/systemd/logind.conf.d/unattended-upgrades-logind-maxdelay.conf is used.

for any value other than 30 seconds ... the Kubernetes implementation of GracefulNodeShutdown is incompatible with the unattended-upgrades package

Environment:

  • Kubernetes version (use kubectl version):
    kubectl version
    Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+IKS", GitCommit:"04e16b9d6c7af21e462f7ac675c34b667cd6149e", GitTreeState:"clean", BuildDate:"2021-05-24T08:21:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+IKS", GitCommit:"04e16b9d6c7af21e462f7ac675c34b667cd6149e", GitTreeState:"clean", BuildDate:"2021-05-24T08:19:13Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration:
    IBM Cloud

  • OS (e.g: cat /etc/os-release):
    cat /etc/os-release
    NAME="Ubuntu"
    VERSION="16.04.7 LTS (Xenial Xerus)"
    ID=ubuntu
    ID_LIKE=debian
    PRETTY_NAME="Ubuntu 16.04.7 LTS"
    VERSION_ID="16.04"
    HOME_URL="http://www.ubuntu.com/"
    SUPPORT_URL="http://help.ubuntu.com/"
    BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
    VERSION_CODENAME=xenial
    UBUNTU_CODENAME=xenial

  • Kernel (e.g. uname -a):
    uname -a
    Linux test-c2v7ef52068f5kndi8s0-testcluster-default-000002f4.iks.ibm 4.4.0-210-generic Less verbose logging for "normal" things. #242-Ubuntu SMP Fri Apr 16 09:57:56 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:

  • Network plugin and version (if this is a network-related bug):

  • Others:

Activity

added
kind/bugCategorizes issue or PR as related to a bug.
on Jun 11, 2021
added
needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
needs-triageIndicates an issue or PR lacks a `triage/foo` label and requires one.
on Jun 11, 2021
kbelow

kbelow commented on Jun 11, 2021

@kbelow
Author

/sig sig-node

k8s-ci-robot

k8s-ci-robot commented on Jun 11, 2021

@k8s-ci-robot
Contributor

@kbelow: The label(s) sig/sig-node cannot be applied, because the repository doesn't have them.

In response to this:

/sig sig-node

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kbelow

kbelow commented on Jun 11, 2021

@kbelow
Author

/sig node

added
sig/nodeCategorizes an issue or PR as relevant to SIG Node.
and removed
needs-sigIndicates an issue or PR lacks a `sig/foo` label and requires one.
on Jun 11, 2021
SergeyKanzhelev

SergeyKanzhelev commented on Jun 11, 2021

@SergeyKanzhelev
Member
pacoxu

pacoxu commented on Jun 24, 2021

@pacoxu
Member
wzshiming

wzshiming commented on Jun 24, 2021

@wzshiming
Member

/assign

mengjiao-liu

mengjiao-liu commented on Jun 24, 2021

@mengjiao-liu
Member

/triage accepted

45 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/bugCategorizes issue or PR as related to a bug.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

Status

Triaged

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @bobbypage@pacoxu@wzshiming@SergeyKanzhelev@rtheis

    Issue actions

      Graceful node shutdown conflicts with unattended-upgrades package · Issue #102818 · kubernetes/kubernetes