Attached PVC vmdks get deleted when a node is removed

[RaceFPV]

When a node is removed via the provider, there are cases where the still attached PVCs can get deleted from disk as well, causing a loss of data and the vmdk file. This then causes k8s to fail to bring up the container on a new node as the underlying PVC is gone.

This can be resolved by having the vm deleted but not deleting all of the PVC vmdk files, or by flagging the PVC vmdks to not delete with vm on creation.

This has cost the loss of data including sql databases etc without warning as there are no prompts or stop telling you what actions will occur on node removal/deletion.

[codenrhoden]

/assign @dvonthenen

[dvonthenen]

@RaceFPV Some clarification... you said "When a node is removed via the provider"... what provider are you talking about?

Trying to understand how the VM is being deleted since we don't have any functionality that deletes VMs.

If you can provide steps to recreate (like using any commands like kubectl or etc), that would be great.

[LinAnt]

Note that this is probably an issue with the in tree cloud-provider.

Drain a node.
Node says it is drained.
The VMDKs remain attached to the node.
If you delete the node the VMDKs are deleted as well, which is expected.

This issue has been present in the in tree cloud provider forever it seems like.

rancher/rancher#15065
rancher/rancher#18221

I think there is a KEP for 1.15 about fixing the drain command :)

[dvonthenen]

This repo is for the out-of-tree or external CCM and CSI driver. I would recommend that you re-file this issue on the k/k repo so the correct people get informed of the bug.

[dvonthenen]

/close

[k8s-ci-robot]

@dvonthenen: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.