Vsphere Cloud Provider: failed to detach volume from shutdown node

[ksandermann]

What happened:
I have a pod with a pv attached to it running on node1.
When I shutdown node1 to simulate node-failure, Kubernetes detects the unhealthy node in the configured timeframe and tries to re-schedule the pod on node2 following the --pod-eviction timeout.
When trying to start the pod on node2, the pv can not be attached as it is still attached to the shutdown node1:

  Warning  FailedAttachVolume      6m               attachdetach-controller  Multi-Attach error for volume "pvc-db44144b-457f-11e9-a7b0-005056af6878" Volume is already exclusively attached to one node and can't be attached to another

Also, the pod on the shutdown node does not get deleted.

What you expected to happen:
As documented here:
The disk should be detached from the shutdown node and attached to the new node where the pod is scheduled on

How to reproduce it (as minimally and precisely as possible):

1. Schedule single pod using pvc on node1
2. Shutdown node1
3. Watch FailedAttachVolume event on the new pod on node2

Anything else we need to know?:
Also, kube-controller-manager does not log anything about this failure.
Detachment and attachment to another nodes works fine, as long as all nodes are healthy.
Force-deletion of the pod on the shutdown node also does nothing.

Environment:

• Kubernetes version (use kubectl version): 1.12.5
• Cloud provider or hardware configuration: vsphere
• OS (e.g: cat /etc/os-release): Ubuntu 16.04
• Kernel (e.g. uname -a): Linux k8s-dev-master3 4.4.0-139-generic On delete, also attempt to update controller state #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Kubespray 2.8.3 using kubeadm
• Others: VSphere 6.5

Thanks in advance! :)

[ksandermann]

/sig vmware

[ksandermann]

@kubernetes/sig-vmware-bugs

[k8s-ci-robot]

@ksandermann: Reiterating the mentions to trigger a notification:
@kubernetes/sig-vmware-bugs

In response to this:

@kubernetes/sig-vmware-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[LinAnt]

It is even worse if you drain a node for an upgrade and then delete the VM. The disks that remain attached will be gone as well as the vm. This is not a recent issue, it has been like this since 1.8.x or 1.9.x.

[yastij]

There's a KEP opened for this kubernetes/enhancements#719

/priority important-soon

[ksandermann]

@yastij I see, thanks for the reference!
Is there any estimated time for this to actually get through?
I see that the PR has been stale for 10 days now

[yastij]

The design is still in discussion, this will be landing on 1.15

[sjberman]

@ksandermann Do you also see an issue where the NotReady node never gets cleaned up? I'm seeing the issue you mentioned, but the powered down node stays in the NotReady state and never goes away. I'm wondering if it has anything to do with the fact that the pod with the volume attached still "exists" on the downed node (though in an "Unknown" state), and these two issues are somehow tied to each other.

[ksandermann]

@sjberman

I didn't test that case, so I can't say anything to that.
What would your expected behaviour be in that case ? That k8s removes the NotReady Node completely from the cluster after time x ?

[sjberman]

@ksandermann Yeah, I would expect a node that is stuck in NotReady for some amount of time to be eventually removed from the cluster. Other cloud providers in different clouds have this behavior.

[zhonglin6666]

isMultiAttachForbidden return true When setting pv with accessMode value 'ReadWriteOnce'，
node1 attach and not detach when node1 down，more than one node attach volume，it will produce errors:
attachdetach-controller Multi-Attach error for volume "pvc-d0fde86c-8661-11e9-b873-0800271c9f15" Volume is already used by pod

// check if this volume is allowed to be attached to multiple PODs/nodes, if yes, return false
for _, accessMode := range volumeSpec.PersistentVolume.Spec.AccessModes {
	if accessMode == v1.ReadWriteMany || accessMode == v1.ReadOnlyMany {
		return false
	}
}

if rc.isMultiAttachForbidden(volumeToAttach.VolumeSpec) {
	nodes := rc.actualStateOfWorld.GetNodesForVolume(volumeToAttach.VolumeName)
	if len(nodes) > 0 {
		if !volumeToAttach.MultiAttachErrorReported {
			rc.reportMultiAttachError(volumeToAttach, nodes)
			rc.desiredStateOfWorld.SetMultiAttachError(volumeToAttach.VolumeName, volumeToAttach.NodeName)
		}
		continue
	}
}

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[yastij]

/remove-lifecycle stale
/lifecycle frozen

[dfsdevops]

I'm also seeing this on 1.18.6, does anyone know any workarounds for this? fwiw I do not see any kind of "NotReady" nodes. I simply scaled the worker nodes down using the TKG cli, most pods got rescheduled successfully, others did not? How can I manually detach a volume?

EDIT: Think I figured it out thanks to info in here: kubernetes-sigs/vsphere-csi-driver#221 (comment)

1. get the name of the pv
2. find the volumeattachment name

kubectl get volumeattachments.storage.k8s.io | grep <pv-name>

3. edit the volumeattachment and remove the finalizer
4. delete the volumeattachment.

kubectl delete volumeattachments.storage.k8s.io <volumeattachment-name>

note: you may also need to ensure that no pods are running and holding onto the volume, so scale down your deployment.

[cheftako]

/assign @andrewsykim
/triage accepted

[k8s-triage-robot]

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Deprioritize it with /priority important-longterm or /priority backlog
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.