VolumeSnapshot deletion behavior

[draghuram]

The page at https://velero.io/docs/v1.11/csi/ mentions this:

"The VolumeSnapshot objects will be removed from the cluster after the backup is uploaded to the object storage, so that the namespace that is backed up can be deleted without removing the snapshot in the storage provider if the DeletionPolicy is `Delete."

Also, the release notes of 1.9.0 at https://github.com/vmware-tanzu/velero/releases/tag/v1.9.0 has this:

"No VolumeSnapshot is left in the source namespace of the workload after a backup finishes."

But this doesn't seem to be the behavior based on my testing using Velero 1.11 and CSI plugin 0.5.0. I see that VolumeSnapshot resources remain in the source namespace after the backup is complete. This happens irrespective of how "deletionPolicy" is configured on the corresponding class.

I will check the code but wanted to post here if anyone else has some info in this regard.

[draghuram]

In today's community call, it was agreed that VolumeSnapshot resources are expected to be removed at the end of the backup so the current behavior is a bug. We will work on a fix.

• Raghu (https://cloudcasa.io).

[anshulahuja98]

@draghuram I tried to repro the issue you described with current @main. But the deletion is happening properly.
Can you double check if anything else is missed.

[draghuram]

We also verified that deletion is happening with 1.9 and 1.10 as expected and are in the process of double checking with newer versions. I tested with 1.11 couple of times and could clearly see VolumeSnapshot resources being left over.

BTW, Can you test with both "Retain" and "Delete" as "deletionPolicy" in VolumeSnapshotClass? I am curious to see what happens in the case of latter. Velero may be patching VolumeSnapshotContent;s deletion policy to "Retain" explicitly before deleting VolumeSnapshot resources but I would like to confirm.

[shubham-pampattiwar]

@draghuram Yes thats the expected behavior, VS is deleted but before deleting the VS, VSC is patched (incase the DeletionPolicy is Delete) and the Deletionpolicy is set to Retain. Here is the code: https://github.com/vmware-tanzu/velero/blob/main/pkg/controller/backup_controller.go#L1059

[anshulahuja98]

We also verified that deletion is happening with 1.9 and 1.10 as expected and are in the process of double checking with newer versions. I tested with 1.11 couple of times and could clearly see VolumeSnapshot resources being left over.

BTW, Can you test with both "Retain" and "Delete" as "deletionPolicy" in VolumeSnapshotClass? I am curious to see what happens in the case of latter. Velero may be patching VolumeSnapshotContent;s deletion policy to "Retain" explicitly before deleting VolumeSnapshot resources but I would like to confirm.

Tried with both deletionPolicies and still able to repro cleanup happening as expected (Latest main)

Can you check your CSI driver logs (if available)
There could be deletion related errors from CSI driver.
Or there could be some finalizers blocking deletion.

IIRC there is also a deletionTimestamp which is added for objects which are marked for deletion. You can check them to validate if velero even tried to trigger delete on them and then CSI driver didn't pick them up.

[draghuram]

Yes, it could be something on my system. I was testing with minikube. I did see the problem last evening but I need to check further to see if there is a finalizer blocking the deletion.

[draghuram]

I tested on two different minikube clusters with 1.11 and see VolumeSnapshot still being left over. Since I enabled debug, I expected to see this message:

"Deleting VolumeSnapshot ..."

but I don't see it. More over, I don't see "deletionTimestamp" in VolumeSnapshot resource so both these facts point to the fact that Velero hasn't tried to delete the resource.

I will debug this further by building main locally.

[draghuram]

The mystery is finally resolved. I used the feature flag "enableCSI" while installing Velero but the actual value should be "EnableCSI". The code that deletes volume snapshots checks if CSI feature is enabled by comparing against "EnableCSI" which, in my case, returns false. So the deletion code is skipped. Once I used the right version of feature name, I could see that volume snapshots are deleted as expected.

I can think of several improvements that can be made:

1. "--enableCSI" should not have been accepted if it is not valid.

2. Since CSI snapshots are created as long as plugin is present, volume snapshot deletion code should not really depend on the feature flag.

[anshulahuja98]

Since CSI snapshots are created as long as plugin is present, volume snapshot deletion code should not really depend on the feature flag.
I think snapshotting also depends on this flag?

[draghuram]

Doesn't seem to be the case. Even when I used incorrect value "enableCSI", snapshots were being created so it doesn't look like tat depends on the feature flag.

[shubham-pampattiwar]

@draghuram I believe CSI snapshots were created because you installed Velero with the CSI plugin

[draghuram]

@shubham-pampattiwar, that is my point. CSI snapshots are created as long as plugin is properly configured. But the deletion of such snapshots depends on the feature flag so there is a disconnect there.