K3s Server Startup Failure

[thiyagarajan-dev]

Environmental Info:
K3s Version:
k3s -v
k3s version v1.33.0+k3s1 (63ab8e5)
go version go1.24.2

Node(s) CPU architecture, OS, and Version:
uname -a
Linux 15117338b0bf 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Tue Jan 14 15:50:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Description:
While starting the k3s server with debug logging enabled and the --snapshotter native flag set, the startup process fails to complete. The attached server log shows multiple informational messages and warnings, ultimately ending with a fatal error that prevents the kubelet from starting successfully.

Impact:
The kubelet fails to launch, preventing the cluster from becoming operational.

Attached:

k3s server --debug --snapshotter=native.txt

[brandond]

v1.33.0+k3s1

Why are you using a version of K3s from April? Can you reproduce this with a more recent patch release?

ERRO[0006] kubelet exited: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory

The problem is obvious from the last line in the log you attached.

I don't know how you ended up on an operating system that is missing /dev/kmsg but you'll need to fix that. If you are running k3s sandboxed or within a container, it needs to be privileged (run as root), and have the full devfs hierarchy available.

There are other errors in the log that suggest that you are attempting to run K3s within an unprivileged VPS:

E0826 09:29:22.537428      17 proxier.go:732] "Error cleaning up nftables rules" err=<
        could not run nftables command: /dev/stdin:1:1-25: Error: Could not process rule: Operation not permitted
        add table ip6 kube-proxy
        ^^^^^^^^^^^^^^^^^^^^^^^^^
 >

[thiyagarajan-dev]

I’m running K3s inside a Docker container with root access using the --privileged flag, so K3s runs as the root user inside the container. Currently using K3s version 1.33.3. The container initially encountered issues due to the missing /dev/kmsg device, and now it's failing because of cgroup v2 compatibility complications.
k3s server --debug --snapshotter=native.txt

E0827 15:37:18.126065 14 node_container_manager_linux.go:62] "Failed to create cgroup" err="cannot enter cgroupv2 "/sys/fs/cgroup/kubepods" with domain controllers -- it is in an invalid state" cgroupName=["kubepods"]
E0827 15:37:18.126102 14 kubelet.go:1688] "Failed to start ContainerManager" err="cannot enter cgroupv2 "/sys/fs/cgroup/kubepods" with domain controllers -- it is in an invalid state"
W0827 15:37:18.126128 14 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods: no such file or directory

k3s server --debug --snapshotter=native

Please see the attached logs for details.

[brandond]

I personally run the k3s image in docker multiple times a day as part of my development workflow, so I know it works. I don't even know how you ended up with missing devfs?

1. you should be able to use the overlayfs snapshotter without issue. Are you using our official k3s image, or are you building your own that does not set things up properly?
2. Are you using hybrid cgroups? You should use either v1 or v2, not hybrid.