restic: ability to run restic restore even when the PV already been restored

[dsuievalov]

Describe the problem/challenge you have
Velero service installed via the official helm chart with restic integration. K8s deployment was update with a proper annotation to make sure that PV will be backuped via restic. velero backup create/velero restore create been run once. Random files were removed from the PV via kubectl exec from the pod to simulate unexpected data loss. velero restore create been run again but the files were not recovered. According to velero logs PV was skipped:

time="2020-01-03T17:38:50Z" level=info msg="Skipping persistentvolumes/pvc-189b3264-2e4c-11ea-b653-42010a8401b9 because it's already been restored." logSource="pkg/restore/restore.go:800" restore=velero/test-restic-20200103193849

Helm installation command:

[~]$ helm upgrade velero vmware-tanzu/velero \
  --install \
  --namespace velero \
  --set configuration.provider=gcp \
  --set-file credentials.secretContents.cloud=<secret.json> \
  --set configuration.backupStorageLocation.name=gcp \
  --set configuration.backupStorageLocation.bucket=<bucket> \
  --set configuration.backupStorageLocation.config.serviceAccount=<serviceAccount> \
  --set configuration.volumeSnapshotLocation.name=gcp \
  --set configuration.volumeSnapshotLocation.config.serviceAccount=<serviceAccount> \
  --set image.repository=velero/velero \
  --set image.tag=v1.2.0 \
  --set image.pullPolicy=IfNotPresent \
  --set initContainers[0].name=velero-plugin-for-gcp \
  --set initContainers[0].image=velero/velero-plugin-for-gcp:v1.0.0 \
  --set initContainers[0].volumeMounts[0].mountPath=/target \
  --set initContainers[0].volumeMounts[0].name=plugins \
  --set-string deployRestic=true \
  --wait --atomic

Backup:

[~]$ velero backup create test-restic --include-namespaces default -l app=nginx -l volume=true --wait

Backup describe: https://gist.github.com/dsuievalov/8048af3f515f1b23dd99225cb0f58c11#file-backup-test-restic-describe

Restore for the first time:

[~]$ kubectl delete deployments/nginx-deployment pvc/nginx-logs
[~]$ velero restore create --from-backup test-restic --wait

Restore describe: https://gist.github.com/dsuievalov/8048af3f515f1b23dd99225cb0f58c11#file-restore-first-time-test-restic-describe

Restore describe after random files removal: https://gist.github.com/dsuievalov/8048af3f515f1b23dd99225cb0f58c11#file-restore-not-first-time-test-restic-describe
Restore logs: https://gist.github.com/dsuievalov/8048af3f515f1b23dd99225cb0f58c11#file-restore-not-first-time-test-restic-logs

Describe the solution you'd like
It would be great if velero restore create can run restic restore even when the PV already been restored.

Anything else you would like to add:

Environment:

• Velero version: v1.2.0
• Kubernetes version: v1.14.7
• Cloud provider or hardware configuration: GKE

[mynktl]

This should be applied to all type of snapshot. Multiple restore on the same PV should be configurable. If restored snapshot/data gets deleted due to some technical issue then only option available is to create a new PV.

[eleanor-millman]

Closing because doesn't fit with the product direction.