kubeadm unmounts /var/lib/kubelet

[rmb938]

What happened:

kubeadm reset unmounts /var/lib/kubelet instead of just emptying the directory

What you expected to happen:

The directory to be emptied instead of unmounted

How to reproduce it (as minimally and precisely as possible):

1. Create a mount point i.e /data/var/lib/kubelet
2. Symlink /var/lib/kubelet to /data/var/lib/kubelet`
3. Launch a cluster with kubeadm init
4. Reset the cluster with kubeadm reset

My setup uses a symlink to the mount point, I am not sure if the bug still happens when /var/lib/kubelet is mounted directly,

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.12.3
• Cloud provider or hardware configuration: bare metal with disks to mount
• OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 (Core)
• Kernel (e.g. uname -a): 4.19.5-1.el7.elrepo.x86_64

/kind bug

[rmb938]

Was asked to post this here kubernetes/kubernetes#71638

[neolit123]

the path is intended for cleanup here:
https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/reset.go#L172

kubeadm reset unmounts /var/lib/kubelet instead of just emptying the directory

yes, the unmount is a pre-step.

My setup uses a symlink to the mount point, I am not sure if the bug still happens when /var/lib/kubelet is mounted directly,

this could be part of the problem, but i don't think it should happen because of symlinks.

[rmb938]

@neolit123 I did some more testing and it seems like this only happens when there are no pod mounts setup.

i.e none of these

tmpfs                                3.9G   12K  3.9G   1% /data/var/lib/kubelet/pods/d8cebd24-f65b-11e8-b24d-000c29aa8cce/volumes/kubernetes.io~secret/kube-router-token-w5r7j
tmpfs                                3.9G   12K  3.9G   1% /data/var/lib/kubelet/pods/d9116102-f65b-11e8-b24d-000c29aa8cce/volumes/kubernetes.io~secret/coredns-token-5djh8
tmpfs                                3.9G   12K  3.9G   1% /data/var/lib/kubelet/pods/d91264dd-f65b-11e8-b24d-000c29aa8cce/volumes/kubernetes.io~secret/coredns-token-5djh8

I ran this command https://github.com/kubernetes/kubernetes/blob/master/cmd/kubeadm/app/cmd/reset.go#L159

and when there are pods mounted it returns:

[root@mac000c29aa8cce ~]# awk '$2 ~ path {print $2}' path=/var/lib/kubelet /proc/mounts
/data/var/lib/kubelet
/data/var/lib/kubelet/pods/d8cebd24-f65b-11e8-b24d-000c29aa8cce/volumes/kubernetes.io~secret/kube-router-token-w5r7j
/data/var/lib/kubelet/pods/d9116102-f65b-11e8-b24d-000c29aa8cce/volumes/kubernetes.io~secret/coredns-token-5djh8
/data/var/lib/kubelet/pods/d91264dd-f65b-11e8-b24d-000c29aa8cce/volumes/kubernetes.io~secret/coredns-token-5djh8

however with no pods mounted it returns this

[root@mac000c29aa8cce ~]# awk '$2 ~ path {print $2}' path=/var/lib/kubelet /proc/mounts
/data/var/lib/kubelet

As you can see it wants to unmount /data/var/lib/kubelet, in the first case it fails because there are bind mounts under it but the 2nd case it success because there is nothing stopping it.

The command probably should be changed to this as it won't contain the mounted path

[root@mac000c29aa8cce ~]# awk '$2 ~ path {print $2}' path=/var/lib/kubelet/ /proc/mounts
/data/var/lib/kubelet/pods/8e3bf290-f65c-11e8-b100-000c29aa8cce/volumes/kubernetes.io~secret/kube-router-token-zvbxp
/data/var/lib/kubelet/pods/8e77f7dd-f65c-11e8-b100-000c29aa8cce/volumes/kubernetes.io~secret/coredns-token-724cf
/data/var/lib/kubelet/pods/8e776f57-f65c-11e8-b100-000c29aa8cce/volumes/kubernetes.io~secret/coredns-token-724cf

There is an extra / at the end of /var/lib/kubelet

[neolit123]

cc @yagonobre @bart0sh

The command probably should be changed to this as it won't contain the mounted path
There is an extra / at the end of /var/lib/kubelet

i think this makes sense, yes.

[bart0sh]

@neolit123 yep, will do

/assign

[strouja]

All, thanks so much. We just encountered this issue on Kubernetes 1.11.5. We are also supporting Kubernetes 1.10.11 installation and it also has this problem. So if you guys can get a patch in for 1.10, 1.11, 1.12, & 1.13 that would be ideal. Thanks again as this was driving us crazy