velero does not create restic repository from manifest if its directories are deleted on s3

[ishbuldin]

What steps did you take and what happened:
There is a configured pod and pvc/pv backup using restic
Backups work, but if you then delete the restic directory in the storage, new backups are broken, without the necessary PV

What did you expect to happen:
Velero must recreate the restic repository in storage according to the resticrepositories manifest

The following information will help us better understand what's going on:

• velero backup describe velero-chart-mybackup-20211202073328

Name:         velero-chart-mybackup-20211202073328
Namespace:    velero-chart
Labels:       app.kubernetes.io/instance=velero-chart
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=velero
              helm.sh/chart=velero-2.26.5
              velero.io/schedule-name=velero-chart-mybackup
              velero.io/storage-location=default
Annotations:  helm.sh/hook=post-install,post-upgrade,post-rollback
              helm.sh/hook-delete-policy=before-hook-creation
              velero.io/source-cluster-k8s-gitversion=v1.18.20
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=18

Phase:  PartiallyFailed (run `velero backup logs velero-chart-mybackup-20211202073328` for more information)

Errors:    1
Warnings:  0

Namespaces:
  Included:  velero-chart
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto

TTL:  15m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2021-12-02 10:35:35 +0300 MSK
Completed:  2021-12-02 10:35:40 +0300 MSK

Expiration:  2021-12-02 10:50:29 +0300 MSK

Total items to be backed up:  68
Items backed up:              68

Velero-Native Snapshots: <none included>

Restic Backups (specify --details for more information):
  Failed:  1

• velero backup logs velero-chart-mybackup-20211202073328

time="2021-12-02T07:35:39Z" level=error msg="Error backing up item" backup=velero-chart/velero-chart-mybackup-20211202073328 error="pod volume backup failed: error running restic backup, stderr=Fatal: unable to open config file: Stat: The specified key does not exist.\nIs there a repository at the following location?\ns3:https://myS3.com/kubernetes-backup-test/restic/velero-chart\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:184" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:435" name=test

Anything else you would like to add:
If you delete the manifest:

API Version:  velero.io/v1
Kind:         ResticRepository

velero will recreate it and recreate directories in the S3 repository

Environment:

• Velero version (use velero version):

Client:
	Version: v1.7.0
	Git commit: 9e52260568430ecb77ac38a677ce74267a8c2176
Server:
	Version: v1.7.0

• Velero features (use velero client config get features):

features: <NOT SET>

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.20", GitCommit:"1f3e19b7beb1cc0110255668c4238ed63dadb7ad", GitTreeState:"clean", BuildDate:"2021-06-16T12:58:51Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.20", GitCommit:"1f3e19b7beb1cc0110255668c4238ed63dadb7ad", GitTreeState:"clean", BuildDate:"2021-06-16T12:51:17Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[ywk253100]

According to the code here, seems only the newly created restic repository can be initialized.

I reproduced this issue in my environment. Tried to delete the existing resource resticrepository and triggered the backup again, but got the following error in restic daemonset:

time="2021-12-06T07:21:25Z" level=error msg="Error running command=restic backup --repo=azure:yinw-blob-container-01:/restic/default --password-file=/tmp/credentials/velero/velero-restic-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=pod=etcd0 --tag=pod-uid=2194540f-b2b7-496d-8c01-ae510d19fd1b --tag=pvc-uid=dd7202e0-e74a-45da-b7c1-ce203e417e8b --tag=volume=etcd0-storage --tag=backup=backup-120603 --tag=backup-uid=23116fc1-ec47-4696-a0fe-70734b33b459 --tag=ns=default --host=velero --json --parent=e6c045a1, stdout=, stderr=Fatal: invalid id \"e6c045a1\": no matching ID found for prefix \"e6c045a1\"\n" backup=velero/backup-120603 controller=pod-volume-backup error="exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/pod_volume_backup_controller.go:291" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*podVolumeBackupController).processBackup" logSource="pkg/controller/pod_volume_backup_controller.go:291" name=backup-120603-z4bxh namespace=velero

Need to do more investigation to check whether this is the expected behavior or not.

Any input is welcome. @ishbuldin

[gwendaljabot]

I also reproduce the issue in similar conditions. Any workaround available ?

[ishbuldin]

I also reproduce the issue in similar conditions. Any workaround available ?

if you are talking about how to create directories in s3

If you delete the manifest:

API Version:  velero.io/v1
Kind:         ResticRepository

velero will recreate it and recreate directories in the S3 repository

it would be better if velero could do it automatically

[gwendaljabot]

Thanks! But, yeah, it would be nice if velero could recreate it automatically.

[RafalMaleska]

@ishbuldin thank you for your workaround. it helped.

velero should definitely recreate the restic repos on s3.

[sureshgoli25]

I also reproduce the issue in similar conditions. Any workaround available ?

if you are talking about how to create directories in s3

If you delete the manifest:

API Version:  velero.io/v1
Kind:         ResticRepository

velero will recreate it and recreate directories in the S3 repository

it would be better if velero could do it automatically
@ishbuldin , kindly help me.
I have not understand the workaround. Do you mean Restic crd to be deleted? or any other

[waldner]

This is still happening with BackupRepository, which I understand is what replaced ResticRepository in recent versions of velero.

[prasoon-pxc]

same for me , any workaround and which crds need to deleted for repo re-initialisation?

[mogaal]

I just faced the same issue (using S3). We should avoid manually deleting the restic folder. Deleting the backup sometimes still leave leftovers in backuprepositories.

The workaround if you deleted the restic folder in the target repository is to delete the backup repository around it.

$ k -n velero delete backuprepositories server-142-sts-default-restic-j94rt
backuprepository.velero.io "server-142-sts-default-restic-j94rt" deleted

This fill force the initialisation again.