kubelet and btrfs snapshots: failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet

[flavio]

Kubernetes version (use kubectl version): 1.5.6 forward

Environment:

• Cloud provider or hardware configuration: both bare metal and virtual
• OS (e.g. from /etc/os-release): SUSE 12+ or openSUSE 42.2+
• Kernel (e.g. uname -a): 4.4.62, but not really relevant in this case
• Install tools: SUSE/openSUSE packages
• Others: none

What happened:

The system is using btrfs as main file system, several btrfs subvolumes are mounted under /, with /var/lib/kubelet being one of them.

cadvisor (and hence kubelet) can't determine the disk space available on the root partition because it gets confused by the subvolume major and minor IDs.

Failed to check if disk space is available on the root partition: failed to get fs info for "root": error trying to get filesystem Device for dir /var/lib/kubelet: err: could not find device with major: 0, minor: 50 in cached partitions map

This is similar to #38337 but fix dc8b6cc is not enough for it.

I've already a fix for this issue, I'll follow up with a PR later today.

What you expected to happen:

No warnings from kubelet.

How to reproduce it (as minimally and precisely as possible):

• Install a system using btrfs as file system for /
• Create a sub-volume for /var/lib/kubelet
• Start kubelet (even in disconnected mode)

Anything else we need to know:

I've seen other similar issues, but none of them was about btrfs. The issues were about tmpfs or zfs. My fix for this issue is going to be based on dc8b6cc hence it's going to be btrfs specific. That's why I filed this new issue.

[k8s-github-robot]

@flavio There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here and label list here

[gnufied]

/sig storage

[jingxu97]

I see the fix for btrfs is merged. Could we close this issue?

[flavio]

Yes, the fix reached Kubernetes too because the vendored version of cadvisor has been upgraded.