kubelet fails to start on k8s v1.10.0-beta.1 / "failed to get rootfs info"

[schu]

Mar 03 11:03:27 kubedee-v1-10-0-beta-1-worker-46igat kubelet[850]: W0303 11:03:27.114392     850 fs.go:539] stat failed on /dev/loop1 with error: no such file or directory
Mar 03 11:03:27 kubedee-v1-10-0-beta-1-worker-46igat kubelet[850]: F0303 11:03:27.114402     850 kubelet.go:1356] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/lib/kubelet": could not find device with major: 0, minor: 110 in cached partitions map

git log v1.9.3..v1.10.0-beta.1 -S RootFsInfo --oneline

shows two commits:

b259543985 collect ephemeral storage capacity on initialization
68dadcfd15 Make eviction manager work with CRI container runtime.

With b259543985 the kubelet fails to start when RootFsInfo returns an error where before it did only log a message. For example from a v1.9.3 kubedee cluster:

Mar 03 11:43:37 kubedee-v1-9-3-worker-ja6ihf kubelet[763]: E0303 11:43:37.233591     763 container_manager_linux.go:583] [ContainerManager]: Fail to get rootfs information failed to get device for dir "/var/lib/kubelet": could not find device with major: 0, minor: 142 in cached partitions map

Two old issues discussing the problem:

• kubelet cannot find device for dir /var/lib/kubelet in cached partitions map kubernetes/kubernetes#38337
• kubelet and btrfs snapshots: failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet kubernetes/kubernetes#47046

..and the corresponding patches:

• https://github.com/google/cadvisor/pull/1574/files
• https://github.com/google/cadvisor/pull/1668/files

In kubedee RootFsInfo fails because /dev/loopX (the mount source of /var/lib/kubelet) was not mounted into the container up until now.