Data lost by k3s-uninstall.sh

[angelnu]

Environmental Info:
K3s Version:

k3s version v1.20.6+k3s1 (8d043282)
go version go1.15.10

Node(s) CPU architecture, OS, and Version:

Linux test-k3s2 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

3 servers

Describe the bug:

The k3s-killall.sh does not unmount all folders under /var/lib/kubelet. Explicitly it does not unmount the CSI ceph mount points which are placed under /var/lib/kubelet/plugins/kubernetes.io/csi/pv. This results on k3s-uninstall deleting their content later on when it does rm -rf /var/lib/kubelet

Steps To Reproduce:

1. Installed K3s
2. Install ceph CSI
3. Deploy a pod with an static cephfs volume (I use a cluster on proxmox baremetal)
4. put some data in the mounted ceph volume
5. k3s-uninstall.sh

Expected behavior:

• All mount under /var/lib/kubeleted being unmounted
• static ceph volume content untouched

Actual behavior:

ceph volumen content lost

Additional context / logs:

NA

[angelnu]

My proposal would be to:

• unmount all /var/lib/kubetet mounts in k3s-killall.sh and not only /var/lib/kubetet/pods
• ensure the rm -rf /var/lib/kubelet does not leave the filesystem (in case a mount point could not be unmounted for any reason)

If needed I could propose a PR.

[angelnu]

@bradtopol - thanks for merging!

Would you consider a backport of this fix to 1.20? I would be happy to trigger a PR there.

[brandond]

@angelnu install.sh is only served off master, and is live as soon as merged - so there's no point in backporting it. You will need to re-run the installer to get the updated uninstall script though.

[angelnu]

I see - I did a test to see if I was getting the fix but it did not work for me. The reason is that I install with Ansible and it turns out that that project keeps a derived copy of install.sh at https://github.com/PyratLabs/ansible-role-k3s/blob/main/templates/k3s-killall.sh.j2

I will do a PR to commit the fix there as well.

Thanks!

Update: opened PyratLabs/ansible-role-k3s#113

[ShylajaDevadiga]

@angelnu Following the steps to reproduce, i am seeing all mounts in /var/lib/kubelet are not unmounted after running k3s-uninstall.

$ ps aux|grep k3s
ubuntu    278043  0.0  0.0   8160   736 pts/0    S+   15:58   0:00 grep --color=auto k3s
$ mount |grep kubelet
devtmpfs on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-df60e9f8-22ac-486a-a3c1-fbba60a48649/dev/6113cbff-7f98-479b-be10-45687c50e6c1 type devtmpfs (rw,relatime,size=2008280k,nr_inodes=502070,mode=755)

During uninstall shows target is busy

+ do_unmount_and_remove /var/lib/kubelet/plugins
+ sort+ awk -v path=/var/lib/kubelet/plugins $2 ~ ("^" path) { print $2 } /proc/self/mounts
 -r
+ xargs -r -t -n 1 sh -c umount "$0" && rm -rf "$0"
sh -c 'umount "$0" && rm -rf "$0"' /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-df60e9f8-22ac-486a-a3c1-fbba60a48649/dev/6113cbff-7f98-479b-be10-45687c50e6c1 
umount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-df60e9f8-22ac-486a-a3c1-fbba60a48649/dev/6113cbff-7f98-479b-be10-45687c50e6c1: target is busy.

[angelnu]

@ShylajaDevadiga - could you please check what process is keeping the mount busy?

I tested with ceph and there after killing all the pods (done a few lines before in killall) the unmount works. Maybe the volumedevices plugin requires additional cleanup.

And for confirmation - did the killall about when hitting the busy error? This should prevent the unexpected delete if the unmount fails.

[ShylajaDevadiga]

@angelnu yes by deleting the pod that uses the pvc, umount is successful.

kubectl delete pod pod-raw

Without deleting the pod here is the fuser results if it helps.

ubuntu@ip-172-31-33-20:~/csi-driver-host-path/examples$ mount |grep plugin
devtmpfs on /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-f8f344e2-360d-43bf-a4f7-d340a9c706cd/dev/6a7c560a-2491-4447-8527-a23ba9124a11 type devtmpfs (rw,relatime,size=2008276k,nr_inodes=502069,mode=755)
ubuntu@ip-172-31-33-20:~/csi-driver-host-path/examples$ sudo fuser -vmM  /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-f8f344e2-360d-43bf-a4f7-d340a9c706cd/dev/6a7c560a-2491-4447-8527-a23ba9124a11
                     USER        PID ACCESS COMMAND
/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-f8f344e2-360d-43bf-a4f7-d340a9c706cd/dev/6a7c560a-2491-4447-8527-a23ba9124a11:
                     root     kernel mount /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-f8f344e2-360d-43bf-a4f7-d340a9c706cd/dev/6a7c560a-2491-4447-8527-a23ba9124a11
ubuntu@ip-172-31-33-20:~/csi-driver-host-path/examples$ 

[angelnu]

if the umount fails then at least some fails are still in use - if it is a process within the container then it should be killed by when the pod is deleted within k3s-killall.sh.

This is why my suggestion is to deck after the mount fail on what process keeps the mount busty with lsof. I suspect that your CSI is starting a process outside the pod hat keeps the mount busy and it is not killed by k3s-killall.sh. Handing for those would process would need to be added if this gets confirmed.

When do cleanly delete the container the CSI does the unmount.

[ShylajaDevadiga]

@angelnu I had use hostpath in the earlier scenario. After internal discussion we decided to use longhorn csi. Validated fix on k3s version v1.21.1+k3s1. Umount was successful.

mount |grep kubelet
...
/dev/longhorn/pvc-c72e462c-81e2-4d37-9a05-456d3aec381f on /var/lib/kubelet/pods/3cd82bb6-1790-403c-966b-fda638ba60ab/volumes/kubernetes.io~csi/pvc-c72e462c-81e2-4d37-9a05-456d3aec381f/mount type ext4 (rw,relatime)