Add support for block volumes

[dzaninovic]

Code marked with "REVIEW:" is either temporary or something needs to be done.
Backup and restore works.

Issue for this code: vmware-tanzu#6548
Design: vmware-tanzu#6590
Interface change: vmware-tanzu#6608

If using Minikube with hostpath CSI driver run this script to fix an issue with creating PVCs from block device snapshot otherwise restore will hang: fix_minikube_loop.tar.gz

Loop devices are not created automatically in the minikube environment so the script will create them ahead of time to workaround the issue.

[dzaninovic]

I fixed the resolveSymlink() issue.

Now Kopia Upload() is failing as it does not support the block devices.
I will apply the same solution as in CloudCasa to resolve it.

[dzaninovic]

I added some minimal Kopia code for block device support and backup completed.
But because Velero runs a backup from the agent container I had to use the elevated privilege for it to be able to open the device.
Not sure if that will be acceptable.
Alternative solution would be to run a backup from the exposer pod but that is a design change.

[dzaninovic]

I added the --privileged-agent option to velero installer.
Since backup works I will start working on the restore and I can come back to the backup after that to optimize it if needed.

[weshayutin]

nice progress, thanks @dzaninovic

[dzaninovic]

I made changes to the restore and restore is now trying to use Kopia and failing as expected:

time="2023-08-03T21:03:12Z" level=warning msg="Failed to patch data download with err Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: error copying: copy file: error creating file: truncate /host_pods/081a1b95-461c-465b-a2b6-bd340f27b347/volumeDevices/kubernetes.io~csi/pvc-d3237e9d-e7a2-45be-8fb2-8ab59c1f0f0c: invalid argument" controller=DataDownload datadownload=restore1-w6w8f error="Failed to run kopia restore: Failed to copy snapshot data to the target: restore error: error copying: copy file: error creating file: truncate /host_pods/081a1b95-461c-465b-a2b6-bd340f27b347/volumeDevices/kubernetes.io~csi/pvc-d3237e9d-e7a2-45be-8fb2-8ab59c1f0f0c: invalid argument" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/restore/local_fs_output.go:153" error.function="github.com/kopia/kopia/snapshot/restore.(*FilesystemOutput.WriteFile" logSource="pkg/controller/data_download_controller.go:323"

I will work on implementing Kopia code to support block device restore.

[dzaninovic]

I refactored the code to adjust for changes in vmware-tanzu#6608.
I also added the restore code and restore worked but I still have to verify if data is restored correctly and if everything works as expected.
I will also see if code can be improved and will rebase with upstream.

[dzaninovic]

I ran a restore test with 10 MB of random data and data was restored with correct md5 hash.
I will work on rebasing this branch with upstream as I see there are some conflicts.

[weshayutin]

nice!

[dzaninovic]

I rebased the branch with upstream, restore still works fine.
Will work on fixing the automated tests.

[dzaninovic]

I merged @shawn-hurley's code and rebased with upstream as there were conflicts.

[dzaninovic]

I reverted #2 since it was causing backup failure.

[dzaninovic]

I fixed all automated tests.

[dzaninovic]

I merged @shawn-hurley's code and tested it. Restored data md5 is correct.

[dzaninovic]

I rebased the branch with upstream and fixed a conflict.

[dzaninovic]

I am working on resolving conflicts caused by merging of vmware-tanzu#6608

[dzaninovic]

I rebased with upstream and made required changes to make it work with vmware-tanzu#6608 changes.
Automated tests are passing.
Restore of regular and block PVC is working as expected and md5 is correct.

Waiting for design PR vmware-tanzu#6590 to be merged before submitting this PR to upstream.

[dzaninovic]

I am ready to create the upstream PR so I will do that after @shawn-hurley is done with his review here.

[dzaninovic]

I created the upstream PR vmware-tanzu#6680 so I will close this.