Can't unmount migrated volumes

[jsafrane]

/kind bug

What happened:
I have a pod with in-tree in-line volume:

apiVersion: v1
kind: Pod
...
spec:
  volumes:
  - name: vsphere-volume-0
    vsphereVolume:
      fsType: ext4
      volumePath: '[WorkloadDatastore] 5137595f-7ce3-e95a-5c03-06d835dea807/e2e-vmdk-1641379409741383902.vmdk'

When CSI migration is enabled, kubelet & the CSI driver is able to run such a pod. When the pod is deleted, its volumes remain mounted + the Pod is Terminating forever:

$ mount | grep 1641379409741383902
/dev/sde on /var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi.vsphere.vmware.com-[WorkloadDatastore] 5137595f-7ce3-e95a-5c03-06d835dea807/e2e-vmdk-1641379409741383902.vmdk/globalmount type ext4 (rw,relatime,seclabel)
/dev/sde on /var/lib/kubelet/pods/00541d84-1c59-4788-bb9b-e97ae5a6ecbe/volumes/kubernetes.io~csi/csi.vsphere.vmware.com-[WorkloadDatastore] 5137595f-7ce3-e95a-5c03-06d835dea807~e2e-vmdk-1641379409741383902.vmdk/mount type ext4 (rw,relatime,seclabel)

What you expected to happen:
The volumes are unmounted.

How to reproduce it (as minimally and precisely as possible):
Enable CSI migration and run in-tree tests for inline vsphere volumes.

Anything else we need to know?:
The root cause is that /proc/mounts and /proc/self/mountinfo escape space " " in the mount path with \040:

$ cat /proc/self/mountinfo | grep sde
7335 6783 8:64 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi.vsphere.vmware.com-[WorkloadDatastore]\0405137595f-7ce3-e95a-5c03-06d835dea807/e2e-vmdk-1641379409741383902.vmdk/globalmount rw,relatime - ext4 /dev/sde rw,seclabel
7337 6783 8:64 / /var/lib/kubelet/pods/00541d84-1c59-4788-bb9b-e97ae5a6ecbe/volumes/kubernetes.io~csi/csi.vsphere.vmware.com-[WorkloadDatastore]\0405137595f-7ce3-e95a-5c03-06d835dea807~e2e-vmdk-1641379409741383902.vmdk/mount rw,relatime - ext4 /dev/sde rw,seclabel

Therefore mount comparison in the driver never finds the volume mounted:

vsphere-csi-driver/pkg/csi/service/osutils/linux_os_utils.go

Line 1013 in 4cbd430

if m.Path == target {

And I can see in the logs:

NodeUnpublishVolume: Target /var/lib/kubelet/pods/00541d84-1c59-4788-bb9b-e97ae5a6ecbe/volumes/kubernetes.io~csi/csi.vsphere.vmware.com-[WorkloadDatastore] 5137595f-7ce3-e95a-5c03-06d835dea807~e2e-vmdk-1641379409741383902.vmdk/mount not present in mount points. Assuming it is already unpublished
NodeUnstageVolume: Target path \"/var/lib/kubelet/plugins/kubernetes.io/csi/pv/csi.vsphere.vmware.com-[WorkloadDatastore] 5137595f-7ce3-e95a-5c03-06d835dea807/e2e-vmdk-1641379409741383902.vmdk/globalmount\" is not mounted. Skipping unstage."

Environment:

• csi-vsphere version: v2.4.0
• vsphere-cloud-controller-manager version: ?
• Kubernetes version: v1.22.1
• vSphere version: 7.0.2
• OS (e.g. from /etc/os-release): RHEL 8
• Kernel (e.g. uname -a): 4.18.0-305.30.1.el8_4.x86_64
• Install tools: OpenShift

[jsafrane]

The CSI driver uses https://github.com/akutz/gofsutil to parse /proc/self/mounts. I filed akutz/gofsutil#9 there, however, it looks abandoned.

[xing-yang]

https://github.com/akutz/gofsutil is no longer maintained. We could consider moving some of the util functions to vsphere-csi-driver repo so that we can fix bugs ourselves.

[jsafrane]

k8s.io/mount-utils provide similar functionality, however, they suffer from the same issue (who would expect spaces in VolumeHandle?)