PersistentVolumeClaims never finish unmounting on Windows nodes

[claudiubelu]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/sig windows

What happened:

After creating pods having PersistentVolumeClaims and deleting them, the logs are getting filled with errors regarding operationExecutor.UnmountDevice failing to unmount the volume [1]. Additionally, tests requiring PVCs will hang several minutes while waiting the PV to unmount [2].

[1] https://paste.ubuntu.com/p/GpJqGMjCQk/
[2] https://paste.ubuntu.com/p/CSfTFNjyzs/

What you expected to happen:

There shouldn't be any issues while unmounting the volume, the tests shouldn't have to wait several minutes waiting for the volume to be unmounted.

How to reproduce it (as minimally and precisely as possible):

1. Create pod requiring PVC.
2. Delete the pod.
3. Check logs for error messages regarding the volume not being able to be unmounted.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.11.2
• Cloud provider or hardware configuration: Azure, acs-engine v0.21.2 ( kubernetes.json: https://paste.ubuntu.com/p/qcTNh5tKfp/ )
• OS (e.g. from /etc/os-release): Windows Server 1803
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[PatrickLang]

/sig azure
/assign @andyzhangx

[andyzhangx]

@BCLAU what's your PVC type? is it azure disk?

[andyzhangx]

@BCLAU thanks for reporting. I have identified that this is a regression introduced by UnmountDevice common code mainly for linux:

kubernetes/pkg/volume/util/operationexecutor/operation_generator.go

Lines 756 to 758 in 426ef9d

	if err != nil || mount.HasMountRefs(deviceMountPath, refs) {
	if err == nil {
	err = fmt.Errorf("The device mount path %q is still mounted by other references %v", deviceMountPath, refs)

the root cause is GetMountRefs is not well implemented

kubernetes/pkg/util/mount/mount_windows.go

Lines 461 to 466 in 426ef9d

	func (mounter *Mounter) GetMountRefs(pathname string) ([]string, error) {
	refs, err := getAllParentLinks(normalizeWindowsPath(pathname))
	if err != nil {
	return nil, err
	}
	return refs, nil

Error code is like following:

E0912 04:11:58.069277    4308 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/azure-disk//subscriptions/xxx/resourceGroups/andy-k8swin1120beta1/providers/Microsoft.Compute/disks/andy-k8swin1120beta1-dynamic-pvc-ec35a4b1-b637-11e8-bb79-000d3af7dab4\"" failed. No retries permitted until 2018-09-12 04:14:00.0692771 +0000 GMT m=+572.648837101 (durationBeforeRetry 2m2s). Error: "GetDeviceMountRefs check failed for volume \"pvc-ec35a4b1-b637-11e8-bb79-000d3af7dab4\" (UniqueName: \"kubernetes.io/azure-disk//subscriptions/xxx/resourceGroups/andy-k8swin1120beta1/providers/Microsoft.Compute/disks/andy-k8swin1120beta1-dynamic-pvc-ec35a4b1-b637-11e8-bb79-000d3af7dab4\") on node \"18986k8s9001\" : The device mount path \"\\\\var\\\\lib\\\\kubelet\\\\plugins\\\\kubernetes.io\\\\azure-disk\\\\mounts\\\\m1433781257\" is still mounted by other references [c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\azure-disk\\mounts\\m1433781257 F:\\]"

I have a different implementation for GetMountRefs func on windows wich could fix this issue, will file a PR tomorrow.