error trying to get filesystem Device for dir /var/lib/kubelet

[Calpicow]

Looks like cadvisor is having trouble getting metrics for /var/lib/kubelet when it is mounted as tmpfs.

Using CentOS 3.10.0-514.6.1.el7.x86_64

kubelet

E0404 21:03:53.256303   17149 kubelet.go:1642] Failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 40 in cached partitions map

/proc/self/mountinfo

122 62 0:40 / /var/lib/kubelet rw,noatime,nodiratime shared:71 - tmpfs tmpfs rw,seclabel

Doesn't appear to be related to #38337 since major/minor numbers match up.

[dims]

Hmm, looks like its coming from here:
https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/google/cadvisor/fs/fs.go#L429

So it's possible that the process needs to be restarted after the directory was mounted? Can you try that? (Since the partitions is cached there)

[Calpicow]

Restarting kubelet gives the same result.

[mcluseau]

I have the same issue under CoreOS Container Linux:

E0407 05:28:05.147388       1 summary.go:97] error while getting metrics summary from Kubelet 10.109.1.4(10.109.1.4:10255): request failed - "500 Internal Server Error", response: "Internal Error: failed RootFsInfo: error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 35 in cached partitions map"

# grep 0:35.*kubelet /proc/self/mountinfo 
462 199 0:35 /var/lib/kubelet /var/lib/kubelet rw,relatime shared:1 - tmpfs tmpfs rw,seclabel

In my case, I use CoreOS's kubelet-wrapper (rkt with the "fly" stage0) and /var/lib/kubelet is bind-mounted rshared before starting rkt.

[mcluseau]

Which seems to be because tmpfs is not in the list here: https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/google/cadvisor/fs/fs.go#L125

[jdalberg]

Any chance of a release supporting tmpfs? I see the same on my rktnetes nodes on Ubuntu.

[0xmichalis]

@kubernetes/sig-node-bugs

[Calpicow]

Fix has been merged to cadvisor, so closing. Thanks all.