Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vsphere cloudprovider does not attempt to detach volumes from shutdown nodes #67900

Closed
gnufied opened this issue Aug 27, 2018 · 19 comments

Comments

@gnufied
Copy link
Member

commented Aug 27, 2018

At least in 1.9, 1.10 and 1.11 release what I am seeing is - vsphere volumes are not getting detached at all when a node is shutdown. Not even when corresponding pods that are using the volumes are deleted.

The reason they are not getting detached is because - when a node is shutdown, it is removed from api-server in vsphere. But the NodeInfo map that vspehre internally maintains needs the node object for successful detach. As a result all detach operations fail with:

ug 24 11:52:10 vim-master.lan atomic-openshift-master-controllers[23519]: E0824 11:52:10.198057   23519 attacher.go:274] Error detaching volume "[datastore1] kubevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk": No VM found
Aug 24 11:52:10 vim-master.lan atomic-openshift-master-controllers[23519]: E0824 11:52:10.198074   23519 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk\"" failed. No 
retries permitted until 2018-08-24 11:52:14.198065767 -0400 EDT m=+3866.556348999 (durationBeforeRetry 4s). Error: "DetachVolume.Detach failed for volume \"pvc-0e972db5-a7b1-11e8-a954-00505694a8ab\" (UniqueName: \"kubernetes.io/vsphere-volume/[datastore1] kubevols/kubernetes-dynamic-
pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk\") on node \"vim-node.lan\" : No VM found"
Aug 24 11:52:10 vim-master.lan atomic-openshift-master-controllers[23519]: E0824 11:52:10.598712   23519 vsphere.go:980] Failed to convert volPaths to devicePaths: map[vim-node.lan:[[datastore1] kubevols/kubernetes-dynamic-pvc-dcfcb0fa-a7ae-11e8-a954-00505694a8ab.vmdk [datastore1] ku
bevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols/kubernetes-dynamic-pvc-178318a1-a7b1-11e8-a954-00505694a8ab.vmdk] vim-master.lan:[[datastore1] kubevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols
/kubernetes-dynamic-pvc-156bdc5c-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols/kubernetes-dynamic-pvc-178318a1-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols/kubernetes-dynamic-pvc-19a4a77a-a7b1-11e8-a954-00505694a8ab.vmdk]]. err: No VM found
Aug 24 11:52:10 vim-master.lan atomic-openshift-master-controllers[23519]: E0824 11:52:10.598727   23519 attacher.go:130] Error checking if volumes are attached to nodes: map[vim-node.lan:[[datastore1] kubevols/kubernetes-dynamic-pvc-dcfcb0fa-a7ae-11e8-a954-00505694a8ab.vmdk [datasto
re1] kubevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols/kubernetes-dynamic-pvc-178318a1-a7b1-11e8-a954-00505694a8ab.vmdk] vim-master.lan:[[datastore1] kubevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] k
ubevols/kubernetes-dynamic-pvc-156bdc5c-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols/kubernetes-dynamic-pvc-178318a1-a7b1-11e8-a954-00505694a8ab.vmdk [datastore1] kubevols/kubernetes-dynamic-pvc-19a4a77a-a7b1-11e8-a954-00505694a8ab.vmdk]]. err: No VM found
Aug 24 11:52:14 vim-master.lan atomic-openshift-master-controllers[23519]: W0824 11:52:14.205293   23519 reconciler.go:235] attacherDetacher.DetachVolume started for volume "pvc-0e972db5-a7b1-11e8-a954-00505694a8ab" (UniqueName: "kubernetes.io/vsphere-volume/[datastore1] kubevols/kub
ernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk") on node "vim-node.lan" This volume is not safe to detach, but maxWaitForUnmountDuration 6m0s expired, force detaching
Aug 24 11:52:14 vim-master.lan atomic-openshift-master-controllers[23519]: E0824 11:52:14.209454   23519 attacher.go:260] Error checking if volume ("[datastore1] kubevols/kubernetes-dynamic-pvc-0e972db5-a7b1-11e8-a954-00505694a8ab.vmdk") is already attached to current node ("vim-node
.lan"). Will continue and try detach anyway. err=No VM found

/sig storage

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2018

/sig vmware

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2018

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2018

@divyenpatel are you saying that in vsphere, the node object no longer gets deleted when a node is shutdown? I ran my tests against k8s-1.9, so it is possible that my build is old and this was changed sometime back. But there are several comments by vmware people that vsphere is unique in this regard that node object gets deleted when a node is powered off.

If that is indeed the case then we will need #67847 to handle the scenario where pods will enter "unknown" state.

@divyenpatel

This comment has been minimized.

Copy link
Member

commented Aug 27, 2018

@gnufied Yes, I see node is not getting deleted.

I can see node is getting listed with the status NotReady, after 30 minutes passed powering off VM.
This is the behavior on the master branch.

# kubectl get nodes
NAME                STATUS                     ROLES     AGE       VERSION
kubernetes-master   Ready,SchedulingDisabled   <none>    45m       v1.13.0-alpha.0.626+ed3c32c3f9af0e-dirty
kubernetes-node1    Ready                      <none>    45m       v1.13.0-alpha.0.626+ed3c32c3f9af0e-dirty
kubernetes-node2    NotReady                   <none>    45m       v1.13.0-alpha.0.626+ed3c32c3f9af0e-dirty
kubernetes-node3    Ready                      <none>    45m       v1.13.0-alpha.0.626+ed3c32c3f9af0e-dirty
kubernetes-node4    Ready                      <none>    45m       v1.13.0-alpha.0.626+ed3c32c3f9af0e-dirty

I see pods on Powered Off Node enters intoUnknownstate.

]# kubectl get pods 
NAME                               READY     STATUS              RESTARTS   AGE
wordpress-mysql-779b4ddc75-wwdmr   1/1       Unknown             0          36m
wordpress-mysql-779b4ddc75-xm5vb   0/1       ContainerCreating   0          28m
@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 27, 2018

Okay this must have changed in some version of k8s, because I can clearly see that is not the case in 1.9. Also this comment by @BaluDontu - #46442 (comment) alludes to same.

Regardless - if that is happening I think #67847 should fix the underlying issue. Can you try pulling that in and see if it works?

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 28, 2018

We should figure out precisely in which version of k8s the node deletion behaviour was changed for vsphere, because we will have to backport #67847 as well to those versions. @divyenpatel would you mind tracking down the k8s version this was changed?

@divyenpatel

This comment has been minimized.

Copy link
Member

commented Aug 28, 2018

Regardless - if that is happening I think #67847 should fix the underlying issue. Can you try pulling that in and see if it works?

Yes this change works along with #67825 for vsphere cloud provider on the master branch.
Refer: #67825 (comment)

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 28, 2018

@divyenpatel pointed me to #63413 PR which was opened to address same problem but was stalled.

cc @SandeepPissay

@SandeepPissay

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2018

We dont have to fix #63413 PR if the node is not getting removed on node shutdown. Do we know on which version of k8s, the node is getting removed on node shutdown?

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2018

I think both me & @divyenpatel have confirmed separately that node does gets removed on shutdown in 1.9 at least. I don't know about later versions or when that change in vsphere was introduced.

@straffalli

This comment has been minimized.

Copy link

commented Sep 14, 2018

I can confirm that node gets removed on shutdown in version 1.11.3. And disk attachement also remains on the poweredOff node.

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Sep 14, 2018

Okay, so I think this bug affects 1.9, 1.10 and 1.11 versions. The key thing is - nothing would detach volumes from these nodes.

@SandeepPissay @divyenpatel I proposed this offline as well. But - will the solution of using FindVmByPath work? If node api object is no longer present in DetachDisk call, we would use node name and folder/path to find the VM as a fallback by iterating through all known datacenters. @divyenpatel pointed out - there could be multiple VMs by same name and if that were the case, we will return error from detach operation and if only one VM by that path/name is found, we proceed with detach logic. Do you think something like that would work?

@SandeepPissay

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2018

@gnufied Please take a look at one of the solution I had in mind. I had proposed this solution here - #63413 (comment), but we could not get to it. It would be nice to solve this problem that can work irrespective of whether vSphere Cloud Provider/Node Manager has the node details in memory or not.

Regarding the solution you proposed, how would it work if the node name is not same as the VM name? What happens if the VM is relocated(storage vMotioned) to a different Virtual Center or Datacenter?

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Sep 24, 2018

The CRD based solution is pretty hard to use and harder to backport to older versions. Configmap based solution may work, so why don't we merge a variant of #63413 ? Are there any pending concerns? We can ask @w-leads if he is interested in updating the PR or one of us can do it.

Also I thought prior to moving to UUID based node search, we required node name to be same name as VM name otherwise IIRC vSphere wasn't able to find the nodes. We anyways use UUID as primary method but can we reliably use it as a fallback or do you think this may cause detaching volumes from incorrect nodes? Also I noticed that nodeInfoMap anyways uses a map of nodeName and NodeInfo - https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/nodemanager.go#L44 , how does that handle case of VM relocation? don't users anyways have to shutdown a node before moving them?

@w-leads

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2018

Hi @gnufied I think #63413 already uses configmap to keep uuid and to detach disks from powered-off nodes. I'm now working on rebasing branch. Do we need to do anything else?

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Sep 24, 2018

@w-leads yeah I know. I was just asking @SandeepPissay if there are any outstanding questions/concerns because of which the PR was never merged. It has been open for such a long time.

@SandeepPissay

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2018

@abrarshivani is looking into #63413 to see if this solves all the use cases for vSphere.

@nikopen

This comment has been minimized.

Copy link
Member

commented Nov 10, 2018

#70291 got merged for 1.11, and says it fixes this one. should #70291 be cherrypicked for 1.10 too? @gnufied

@gnufied

This comment has been minimized.

Copy link
Member Author

commented Nov 10, 2018

@nikopen it was cherry picked to 1.10 - #70808

@gnufied gnufied closed this Nov 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.