Pod fail to start as /sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice doesn't exist

[trashadewan]

What happened?

Pod failed to start with the following issue:

Warning FailedCreatePodContainer 29m kubelet unable to ensure pod container exists: failed to create container for [kubepods burstable pod71ab4a03-d88b-4cbb-bd26-41c5b0ce3663] : Timeout waiting for systemd to create kubepods-burstable-pod71ab4a03_d88b_4cbb_bd26_41c5b0ce3663.slice

What did you expect to happen?

The pod to be successfully created and complete after running.

How can we reproduce it (as minimally and precisely as possible)?

I am unsure. once i reboot the node, the issue is resolved and the pods can start on it.

Anything else we need to know?

When i see logs from journalctl -u kubelet:
I see a lot of pods logs like below:

Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974117  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/ab2b8cef-f0fb-4de0-ab3a-3e04cde0f37d/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974152  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/d656b533-ebb5-408a-a515-77ebb6d11217/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974184  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/fe8c66f0-4aa5-4784-9585-a9d093b977fe/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974212  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/52ca3ffa-54d4-4dc8-b1be-df53183faa74/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974243  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/8f258a92-843f-45bb-a8ec-372c796f1205/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974275  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/dec19472-1e63-476c-b66f-7e2a1f8b8501/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974302  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/47002ae2-3a51-44c0-9d4c-51a469cac463/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974334  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/5ede6e21-4159-4736-857e-3346b8899bb3/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974367  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/a587eec9-42d8-43ff-9491-0db1ceefb296/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974396  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/b384e091-a24e-4846-91b0-5255fb301718/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974424  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/ed7a5c77-70e8-4a6d-8363-6df0177f492c/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974467  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/0c7305e5-b30b-4330-8112-a88aad9d9361/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974496  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/4d777a20-2f14-4efc-ad4e-1aee46b207de/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974523  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/f67c5158-8129-4db3-bb1c-2ec408a0f498/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974553  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/3a6b8f57-32a8-42bd-bb56-4f858e154cc1/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974600  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/a9d40fe7-fdbd-4d3e-a3f7-fc1435ad09c0/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974632  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/996f6de3-b63c-492c-8597-704e9338958e/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974658  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/b55779a4-4c59-4cdd-8584-7785df73e7db/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974686  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/c3942c0a-2c76-4a95-8ae5-c8d7cca797d1/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974719  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/005d6b12-57d2-4779-b02a-8c9e5611b381/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974756  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/2ffbb08f-f3f6-40ea-80c9-9c38d130a282/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974764  160756 pod_container_manager_linux.go:191] "Failed to delete cgroup paths" cgroupName=[kubepods burstable pod939ec36a-261c-46c4-80b4-af638d670e05] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod939ec36a-261c-46c4-80b4-af638d670e05] : Timed out while waiting for systemd to remove kubepods-burstable-pod939ec36a_261c_46c4_80b4_af638d670e05.slice"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974785  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/30355c4a-55ad-4bf9-afa4-84ee5fe834ca/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974816  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/73dd1ad7-32ea-4ab3-a666-6745f607be33/volumes"
Feb 22 15:16:14 isaac-hil-ovx-07 kubelet[160756]: I0222 15:16:14.974843  160756 kubelet_getters.go:306] "Path does not exist" path="/var/lib/kubelet/pods/c952e257-ae07-4389-b177-d5f117346801/volumes"

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

local on prem cluster

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ uname -a
# Linux isaac-hil-ovx-07 5.15.0-86-generic #96~20.04.1-Ubuntu SMP Thu Sep 21 13:23:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

</details>


### Install tools

<details>
runc --version
runc version 1.1.9
commit: v1.1.9-0-gccaecfc
spec: 1.0.2-dev
go: go1.20.8
libseccomp: 2.5.1


</details>


### Container runtime (CRI) and version (if applicable)

<details>
containerd --version
containerd containerd.io 1.6.24 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[neolit123]

/sig node
for triage

version.Info{Major:"1", Minor:"25"

this version is not in support
https://kubernetes.io/releases/

[AnishShah]

@trashadewan, it is difficult to fix the issue without repro step. Can you provide any repro steps or logs?

/triage needs-information
/priority backlog

[Sivakajan-tech]

@trashadewan Can you provide the repro steps?

[trashadewan]

I wasn't able to reproduce the issue at all Upgraded to 22.04 and since then didn't see the issue Sent from Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Sivakajan Sivaparan ***@***.***> Sent: Tuesday, March 12, 2024 2:02:56 AM To: kubernetes/kubernetes ***@***.***> Cc: Trasha Dewan ***@***.***>; Mention ***@***.***> Subject: Re: [kubernetes/kubernetes] Pod fail to start as /sys/fs/cgroup/devices/kubepods.slice/kubepods-burstable.slice doesn't exist (Issue #123459) @trashadewan<https://github.com/trashadewan> Can you provide the repro steps? — Reply to this email directly, view it on GitHub<#123459 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABYYS3FNPKJW7CWTXRMLKW3YX3AEBAVCNFSM6AAAAABDV4SIOWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJRGA4DMNJYHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[haircommander]

If there's no reproducer, and folks haven't caught it on newer versions, then I'm inclined to

/close

please reopen if you hit this on a supported k8s version

[k8s-ci-robot]

@haircommander: Closing this issue.

In response to this:

If there's no reproducer, and folks haven't caught it on newer versions, then I'm inclined to

/close

please reopen if you hit this on a supported k8s version

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.