Kopia - Failed to restore more than 11 PVs , Error: "Unable to load snapshot {ID}: snapshot not found"

[duduvaa]

What steps did you take and what happened:

Running kopia backup & restore.
Create ns with 5,10,20 & 30 Pods & PVs.
All backups were passed with 'Completed' status.
While running restore:
ns with 5 & 10 PVs - 'Completed' status. All Pods with 'Running' status
ns with 20 & 30 PVs - 'PartiallyFailed' status, only 11 Pods with 'Running' status, the rest Pods with 'Init:0/1' status.
All PVCs are with 'Bound' status.

Steps & Commands:

1. Creare namespaces with 5,10,20 & 30 Pods&PVs. (Each PV size is 6GB with data usage 2GB.)
2. Backup the 4 namespaces
   ./velero backup create kopia-backup-5pods-2gb --include-namespaces perf-busy-data-cephrbd-5pods-2gb --default-volumes-to-fs-backup --snapshot-volumes=false -nupstream-velero
3. Delete the namespaces
4. Running Restore
   ./velero restore create kopia-restore-5pods-2gb --from-backup kopia-backup-5pods-2gb -nupstream-velero

restore of ns with 5 & 10 PVs - Completed
restore of ns with 20 & 30 Pvs - PartiallyFailed.

NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
kopia-restore-10pods-2gb kopia-backup-10pods-2gb Completed 2023-09-03 08:07:42 +0000 UTC 2023-09-03 08:13:24 +0000 UTC 0 10 2023-09-03 08:07:42 +0000 UTC
kopia-restore-20pods-2gb kopia-backup-20pods-2gb PartiallyFailed 2023-09-03 08:24:37 +0000 UTC 2023-09-03 08:31:16 +0000 UTC 9 10 2023-09-03 08:24:37 +0000 UTC
kopia-restore-30pods-2gb kopia-backup-30pods-2gb PartiallyFailed 2023-09-03 08:33:22 +0000 UTC 2023-09-03 08:36:23 +0000 UTC 19 10 2023-09-03 08:33:22 +0000 UTC
kopia-restore-5pods-2gb kopia-backup-5pods-2gb Completed 2023-09-03 08:01:39 +0000 UTC 2023-09-03 08:06:15 +0000 UTC 0 10 2023-09-03 08:01:39 +0000 UTC

Errors:
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 12f5d4eaa6f8f7dcd1c7c330adc1a07f: snapshot not found
kopia-restore-20pods-2gb-describe.txt: pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot f29e0d7bd0e89a0e690505032f9f5184: snapshot not found
kopia-restore-20pods-2gb-describe.txt: pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 176ef4dc665a8bd8375ef78a6f08c318: snapshot not found

time="2023-09-03T08:25:02Z" level=error msg="unable to successfully complete pod volume restores of pod's volumes" error="pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 12f5d4eaa6f8f7dcd1c7c330adc1a07f: snapshot not found" logSource="pkg/restore/restore.go:1731" restore=upstream-velero/kopia-restore-20pods-2gb
kopia-restore-20pods-2gb.log:time="2023-09-03T08:25:03Z" level=error msg="unable to successfully complete pod volume restores of pod's volumes" error="pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot f29e0d7bd0e89a0e690505032f9f5184: snapshot not found" logSource="pkg/restore/restore.go:1731" restore=upstream-velero/kopia-restore-20pods-2gb

What did you expect to happen:
Restore all pods & PVs successfully

The following information will help us better understand what's going on:

Attached Velero debug of all 4 restores.
kopia-restore-5pods-2gb.tar.gz
kopia-restore-30pods-2gb.tar.gz
kopia-restore-20pods-2gb.tar.gz
kopia-restore-10pods-2gb.tar.gz

Anything else you would like to add:

Pod with 'Init:0/1' status:

oc describe pod busy-data-rbd-30pods-2gb-16-b77d9cd98-rb6fl -nperf-busy-data-cephrbd-30pods-2gb

Init Containers:
restore-wait:
Container ID: cri-o://68f159b3a97d59aed6f7cc2c12417041bc8a96bf41432f00c9927d3dcae9e5ef
Image: velero/velero-restore-helper:main
Image ID: docker.io/velero/velero-restore-helper@sha256:2ddad48ce9bbbbc965bf8a27d70cb9c303aa31c10f0ce06eab9a72f7399adaf7
Port:
Host Port:
Command:
/velero-restore-helper
Args:
3e44e282-2e2a-4082-87fc-52f13f5afbfe
State: Running
Started: Sun, 03 Sep 2023 08:33:53 +0000
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Environment:
POD_NAMESPACE: perf-busy-data-cephrbd-30pods-2gb (v1:metadata.namespace)
POD_NAME: busy-data-rbd-30pods-2gb-16-b77d9cd98-rb6fl (v1:metadata.name)
Mounts:
/restores/vol-0 from vol-0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xwftf (ro)

oc logs pod/busy-data-rbd-30pods-2gb-1-c945fcd9d-dqkqv -c restore-wait -nperf-busy-data-cephrbd-30pods-2gb

Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe

oc logs pod/busy-data-rbd-30pods-2gb-1-c945fcd9d-dqkqv -c restore-wait -nperf-busy-data-cephrbd-30pods-2gb | grep -c "Not found"

20883

oc logs pod/busy-data-rbd-30pods-2gb-1-c945fcd9d-dqkqv -c restore-wait -nperf-busy-data-cephrbd-30pods-2gb > /root/restore-wait

restore-wait.gz

Pod with 'Running' status:

oc describe pod busy-data-rbd-30pods-2gb-7-66fbb475f6-d2v5l -nperf-busy-data-cephrbd-30pods-2gb

Init Containers:
restore-wait:
Container ID: cri-o://fc1ed2b6adf59ee6395cd420ffcb19e3b64ab054d5cf644bf063d7638f1fd376
Image: velero/velero-restore-helper:main
Image ID: docker.io/velero/velero-restore-helper@sha256:3c01a8c5efcc7374481b21f19dfe949724ca5149801af549dd64e2b3a2cc2d95
Port:
Host Port:
Command:
/velero-restore-helper
Args:
3e44e282-2e2a-4082-87fc-52f13f5afbfe
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 03 Sep 2023 08:34:11 +0000
Finished: Sun, 03 Sep 2023 08:34:42 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 100m
memory: 128Mi
Environment:
POD_NAMESPACE: perf-busy-data-cephrbd-30pods-2gb (v1:metadata.namespace)
POD_NAME: busy-data-rbd-30pods-2gb-7-66fbb475f6-d2v5l (v1:metadata.name)
Mounts:
/restores/vol-0 from vol-0 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mhv2f (ro)

#oc logs pod/busy-data-rbd-30pods-2gb-7-66fbb475f6-d2v5l -c restore-wait -nperf-busy-data-cephrbd-30pods-2gb
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Not found: /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe
Found /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfeAll restic restores are done
unlinkat /restores/vol-0/.velero/3e44e282-2e2a-4082-87fc-52f13f5afbfe: permission denied

oc logs pod/busy-data-rbd-30pods-2gb-7-66fbb475f6-d2v5l -c restore-wait -nperf-busy-data-cephrbd-30pods-2gb | grep -c "Not found"

30

Environment:

• Velero version: main (Velero-1.12) , last commit
  commit 30e54b0 (HEAD -> main, origin/main, origin/HEAD)
  Author: Daniel Jiang jiangd@vmware.com
  Date: Wed Aug 16 15:45:00 2023 +0800

• Velero features (use velero client config get features):

  ./velero client config get features

  features:

• Kubernetes version (use kubectl version):

  oc version

Client Version: 4.12.9
Kustomize Version: v4.5.7
Server Version: 4.12.9
Kubernetes Version: v1.25.7+eab9cc9

• Cloud provider or hardware configuration:
  OCP running over BM servers
  3 masters & 6 workers nodes

  oc get nodes

NAME STATUS ROLES AGE VERSION
master-0 Ready control-plane,master 148d v1.25.7+eab9cc9
master-1 Ready control-plane,master 148d v1.25.7+eab9cc9
master-2 Ready control-plane,master 148d v1.25.7+eab9cc9
worker000-r640 Ready worker 148d v1.25.7+eab9cc9
worker001-r640 Ready worker 148d v1.25.7+eab9cc9
worker002-r640 Ready worker 148d v1.25.7+eab9cc9
worker003-r640 Ready worker 148d v1.25.7+eab9cc9
worker004-r640 Ready worker 148d v1.25.7+eab9cc9
worker005-r640 Ready worker 148d v1.25.7+eab9cc9

• OS (e.g. from /etc/os-release):
  Red Hat Enterprise Linux CoreOS 412.86.202303211731-0
  Part of OpenShift 4.12, RHCOS is a Kubernetes native operating system

cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION="412.86.202303211731-0"
VERSION_ID="4.12"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 412.86.202303211731-0 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://docs.openshift.com/container-platform/4.12/"
BUG_REPORT_URL="https://access.redhat.com/labs/rhir/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.12"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.12"
OPENSHIFT_VERSION="4.12"
RHEL_VERSION="8.6"
OSTREE_VERSION="412.86.202303211731-0"

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[Lyndon-Li]

@duduvaa You don't see the same problem for data mover, right?

[duduvaa]

@Lyndon-Li , right
datamover restore had other issue (#6733)

[Lyndon-Li]

Here are all the errors I found in the attached logs, 28 in total:

time="2023-09-03T08:33:55Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot ab108cb11c2f2d3bfa3db32a00dac5db: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-g2xrp
time="2023-09-03T08:33:56Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot ec921c7b01e296d0c60a3894e85e75d9: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-cgl8g
time="2023-09-03T08:34:11Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 639e3f335367e33c9eca6b723527acc6: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-hfvkf
time="2023-09-03T08:34:12Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 99e24644e8824b0fadccbcbc973c06a1: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-kj9lj
time="2023-09-03T08:33:52Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 2a6336ed4378f52a088b049f683e7e9f: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-jxxjn
time="2023-09-03T08:34:02Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot c60f9ef56d07954fd3d82319ad219fa9: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-wphjz
time="2023-09-03T08:34:53Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 7946b78d7cd273167c21c969ad7ffc14: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-znnnt
time="2023-09-03T08:25:02Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 12f5d4eaa6f8f7dcd1c7c330adc1a07f: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-hnn5l
time="2023-09-03T08:25:03Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot f29e0d7bd0e89a0e690505032f9f5184: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-4cd2h
time="2023-09-03T08:25:10Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 176ef4dc665a8bd8375ef78a6f08c318: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-qvwgm
time="2023-09-03T08:26:03Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 4b2f59657d5655b54d2e7322a3274e5b: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-4f7jl
time="2023-09-03T08:27:15Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot e54f3cfb24e8a7713acf40f025170a33: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-htx88
time="2023-09-03T08:33:54Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 419cf125fa7039afd7c37499a2d1625e: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-fnrhk
time="2023-09-03T08:34:04Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot ff7964cc393503825ef86b894018e7a3: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-7bxnw
time="2023-09-03T08:34:05Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 19424efa6c179e92d67d63a3a3774da5: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-x5z72
time="2023-09-03T08:34:13Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 790dae21bc7128632bcc6d27cea74a88: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-t449k
time="2023-09-03T08:35:05Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 5f82438eaa4e80238cc752d384268601: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-8qbjc
time="2023-09-03T08:26:11Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 3ec697fb182644f8a81b94a6a608e0f7: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-pnjzj
time="2023-09-03T08:26:12Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 95e9a4c3d27a332145cfbd3c95a22685: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-8d5jg
time="2023-09-03T08:28:13Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 57e56838926a3dc255851599306042ff: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-s7mcm
time="2023-09-03T08:31:16Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 939950a30780c53aee4575b064e24951: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-20pods-2gb-dcxk5
time="2023-09-03T08:33:52Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 49fbb3062dd2dbc29cf7006f8718474c: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-xw469
time="2023-09-03T08:34:03Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot e19a6f279b731e0ad16ab7b75666b5db: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-sx5dh
time="2023-09-03T08:34:11Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 7f4e8257a64d3ff034cf46f9b6fd0bca: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-cxr6s
time="2023-09-03T08:34:12Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 74ada7bd8e9d07b84ba8523e9c4395ee: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-mb9sk
time="2023-09-03T08:34:14Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 2450b834d69275d35198feb0fd917ff7: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-vl7rw
time="2023-09-03T08:35:04Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 66b3560e50aafba9395cb39da464d0ef: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-tjqzd
time="2023-09-03T08:36:05Z" level=error msg="Async fs restore data path failed" controller=PodVolumeRestore error="Failed to run kopia restore: Unable to load snapshot 0b9c138d1fbff2249984b1a6b599f642: snapshot not found" error.file="/go/pkg/mod/github.com/kopia/kopia@v0.13.0/snapshot/manager.go:26" error.function=github.com/kopia/kopia/snapshot.init logSource="pkg/controller/pod_volume_restore_controller.go:326" pvr=kopia-restore-30pods-2gb-j58cv

[Lyndon-Li]

@duduvaa
Could you help to check the existence of those snapshot in the Kopia repo by running below Kopia command?
kopia manifest list | grep <snapshotID>

"snapshotID" should be one of the snapshot IDs list above.

BTW, please use Kopia 0.13 binary and before running kopia manifest list, kopia repository connect needs to be run first.

[duduvaa]

@Lyndon-Li ,
The 28 items list are belong to failed PVs:
ns-20 pods - 9 errors
ns-30 pods - 19 errors

none of the "snapshotID" exists in "manifest list"

The below output shows the successful restore

namespace "perf-busy-data-cephrbd-20pods-2gb" - 11 ids - match to succeed restore:

./kopia manifest list

010a01993b0934f019805f6ae347d426 213 2023-08-31 15:59:59 UTC type:maintenance
01dba7613b35412e1c49afe73bc305a0 1052 2023-08-31 16:02:10 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:93c0233c-f092-417f-989e-53e971679fe3 pod:busy-data-rbd-20pods-2gb-18-6df7459bdf-k7jbq pvc-uid:a52c0602-42fa-45f3-82fa-fd2a276137d0 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
6518781670c4c104c41c1b8b70060320 1052 2023-08-31 16:02:23 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:fbc97f52-78d2-4691-9d49-8835a8a8f893 pod:busy-data-rbd-20pods-2gb-19-79dfd95668-cttmp pvc-uid:c2660ef5-30aa-4e52-8ad4-8ab4c3fa636b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
dd63cbc26675f99379d75c1492bbff19 1052 2023-08-31 16:02:36 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:8bf8b043-f5e0-49c6-8d7e-7062ba4f8b2e pod:busy-data-rbd-20pods-2gb-2-7d6d6c55cd-cjx6x pvc-uid:14ef8302-bbc9-4b95-9302-17133bfe7ba9 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
aa969040a2cbb8548868c6c6122b2e67 1053 2023-08-31 16:02:49 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:ac1ccb7a-a8bc-4a36-9713-4695ba2573f4 pod:busy-data-rbd-20pods-2gb-20-6dd7cd8496-n7cnv pvc-uid:41c40d51-b020-443c-9bfa-e795c313858b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
9003be0534a02efe1c8dddb000d0a69a 1052 2023-08-31 16:03:02 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:a051dd4f-83bc-4b25-9324-9dc5b47b3463 pod:busy-data-rbd-20pods-2gb-3-584499c959-xqhkw pvc-uid:a39d31bf-c470-48db-a2bd-f43706673d0c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f28df4e791a8e4771a35adf7e28d8bc7 1052 2023-08-31 16:03:15 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:53257d44-11f5-4aac-829a-87751ea324b4 pod:busy-data-rbd-20pods-2gb-4-7c8b567d84-q5xtl pvc-uid:0c3cbdf6-7fcb-4b53-b5b2-1efc6d157b7c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f9ab5670efee51c8756586be98e7cb29 1053 2023-08-31 16:03:28 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:3b4714d8-38f1-4a11-b618-2e54841a411a pod:busy-data-rbd-20pods-2gb-5-7c7ccfbd5c-htmgf pvc-uid:880f924b-6e4b-4b54-8462-0cfc48d5c022 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
1563d949282f980558572bbf60f3d321 1052 2023-08-31 16:03:42 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:5f4a13c3-9f9a-4c57-95c5-e3aa661c09cc pod:busy-data-rbd-20pods-2gb-6-575f8db5fd-sb8c7 pvc-uid:5549f66d-553f-43a0-a644-8b3afb71f271 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
248925db2b78baa448b4ed2d79eedce7 1052 2023-08-31 16:03:55 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:ad81dec3-e988-461e-b3a1-f38c32c1ecc1 pod:busy-data-rbd-20pods-2gb-7-7f7468bc6c-kdtkt pvc-uid:9a64c98f-c53c-4d69-8248-634222da4a86 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
580a36d8da3b5d13ca57ac552cc86084 1052 2023-08-31 16:04:08 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:a0bd7276-b8ac-4572-82c8-618cd8dcd807 pod:busy-data-rbd-20pods-2gb-8-6c496c4776-6rd66 pvc-uid:58c80880-edf5-4293-bb50-948eccfe04f5 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
d12d773b30232d372803393fcf2735f0 1052 2023-08-31 16:04:22 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:9e92c684-419e-4380-a702-a20fc7ac0766 pod:busy-data-rbd-20pods-2gb-9-5bcf597b6f-5kmnr pvc-uid:9c27dce5-ccf8-46bf-bda5-f82eed6be03b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0

namespace "perf-busy-data-cephrbd-30pods-2gb" - 11 ids - match to succeed restore:

./kopia manifest list

561b1374fded9774751ebbd35cba46ac 213 2023-08-31 16:14:50 UTC type:maintenance
6f8a73acbbb71f272ed3382f9edc2ee4 1053 2023-08-31 16:19:18 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:3b506188-1c9a-44d8-b5da-03ae1619a212 pod:busy-data-rbd-30pods-2gb-27-5bcd8f6c4b-cjdmt pvc-uid:e864ce2d-5e01-4e0e-bf98-63567425cf80 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
a7b31b3194156dcc28315981832ffdd5 1053 2023-08-31 16:19:31 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:f01035df-b786-4f60-b708-ee370af84d50 pod:busy-data-rbd-30pods-2gb-28-86c4fd57cc-f78k4 pvc-uid:7b6bdc97-36a6-42dd-8fc7-6ff0f16cca5f snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
20ea9b490eb08fcc518b0ac545d26ba9 1053 2023-08-31 16:19:44 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:0cd4c0e6-01bf-4d29-953d-06dd21702f23 pod:busy-data-rbd-30pods-2gb-29-8677d44d5b-zp6mv pvc-uid:a3a3472e-ba1a-4deb-8143-a71c0071eb86 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
cbb880709a5b193ca3cea031741236fc 1051 2023-08-31 16:19:57 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:ea505fd4-a436-4055-b97f-6683248d63a2 pod:busy-data-rbd-30pods-2gb-3-668cf4fbcb-tzxk6 pvc-uid:c3ebb995-686b-4229-a25c-c08b59b4a1e9 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f02cadae1f3476b3b3d409941bc3f677 1052 2023-08-31 16:20:10 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:5cccb406-050f-415b-84c6-033664d82483 pod:busy-data-rbd-30pods-2gb-30-7f9896ccc4-m64jl pvc-uid:453045be-c5cb-4376-8bc4-22458865b32b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
813ec141a73037c5ca034b852137c0ea 1052 2023-08-31 16:20:23 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:ddf0ae7d-491c-4a00-9829-2fe04df55059 pod:busy-data-rbd-30pods-2gb-4-69cb876667-2tt7r pvc-uid:88bfac46-aaf2-491d-8348-cfbf7ee3ea0c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
11a477915f711172b42949ea0e098915 1051 2023-08-31 16:20:36 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:429746ee-bd0f-4410-8870-91e85088c9b8 pod:busy-data-rbd-30pods-2gb-5-675ffd8874-zg8s2 pvc-uid:fb183164-b7a8-46d7-a885-b7b38ef8f685 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
fc746cc61b5f0bad13dc324409188fd2 1051 2023-08-31 16:20:50 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:853e132d-2cad-4f55-a9a9-1ff3a3803fc0 pod:busy-data-rbd-30pods-2gb-6-59bb4576df-dkfbn pvc-uid:c69ee6fa-b9c1-40f2-8f38-16c4ef1e8199 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
35e793613e318d1f01b49ee2ab381aa8 1050 2023-08-31 16:21:03 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:a5be57c8-429a-40a4-8373-563490f14438 pod:busy-data-rbd-30pods-2gb-7-66fbb475f6-d2v5l pvc-uid:b08108a4-c3da-4bd8-8cd8-b8f3f494f37c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
36132b36e8acbcc14afb1037a9c87583 1052 2023-08-31 16:21:16 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:cff0822b-43d7-4fd5-9d8f-82ffeb9e3b79 pod:busy-data-rbd-30pods-2gb-8-76f7df79b7-b5ghf pvc-uid:38905cc7-9abf-4317-a4e4-f8657d6de4df snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
251ee27b63faa7a5da6f4388a8aa399e 1052 2023-08-31 16:21:30 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:23409050-0f5e-4a3d-ae81-3cb206775b55 pod:busy-data-rbd-30pods-2gb-9-7c9bb6fb6b-p2n58 pvc-uid:03f7a2a5-d27e-436e-aeef-740cfe3dd677 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0

[Lyndon-Li]

@duduvaa
We need to check whether we can find the snapshot IDs for the 28 failed ones, as listed here #6748 (comment).
Please use command kopia manifest list | grep <snapshotID> to test against these IDs.
Please also run kopia snapshot list to see how many snapshots have been identified.

[duduvaa]

./kopia snapshot list

No snapshots found. Pass --all to show snapshots from all users/hosts.

./kopia snapshot list --all
default@default:.
2023-08-31 16:19:05 UTC k9ac51c064280785121f3bf9f16f83b20 2.1 GB dgrwxrwxr-x files:6901 dirs:33
2023-08-31 16:19:19 UTC k0cfbaf3d970dc089e1343b5d966caa72 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-10)
2023-08-31 16:19:32 UTC k81a153c76aa4d0f0ef818f81d40dcec9 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-9)
2023-08-31 16:19:45 UTC ka171e58e54b445c5085adeb5a93223af 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-8)
2023-08-31 16:19:58 UTC k0b53576a0d3341d7e0dcd3ac0d74c9cd 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-7)
2023-08-31 16:20:11 UTC k61661b05de336d17c1126f3d19edbbdf 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-6)
2023-08-31 16:20:25 UTC k2f96a92090290612aa607eda5744836e 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-5)
2023-08-31 16:20:38 UTC kddd5e1b5f917f3d4e91bebcdaf010a55 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-4)
2023-08-31 16:20:51 UTC k34e2903aa38bffd4ec360656183d0380 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-3)
2023-08-31 16:21:05 UTC k7760b4e0af268a5a89f82659151f296f 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-2)
2023-08-31 16:21:18 UTC k69019e0be52fe98444efc2673cd8245d 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

[Lyndon-Li]

The problem was caused by the misuse of Kopia default retention policy. As a result, for one source, only 10 latest snapshots are preserved, as the number in Kopia default retention policy is 10.

Velero has its own retention policy on the backup level, the snapshots saved in the Kopia repo should follow the backup retention policy instead of using Kopia's default policy.

This is a regression due to code changes in 1.12. Both fs-backup and data mover are affected.
Moreover, we need to remove the code that deletes snapshots by checking Kopia's retention policy for the same reason.

[duduvaa]

1st namespace

kopia-backup-20pods-2gb Completed 0 0 2023-08-31 15:59:52 +0000 UTC 25d default
kopia-restore-20pods-2gb kopia-backup-20pods-2gb PartiallyFailed 2023-09-03 08:24:37 +0000 UTC 2023-09-03 08:31:16 +0000 UTC 9 10 2023-09-03 08:24:37 +0000 UTC

./kopia repository connect s3 --bucket="oadp-bucket" --access-key=Cvaz1RguhpAyuAYnAETQ --secret-access-key=+BSgIgkn0EvUT+kSeXcH8u4xo6MoHzSVRPG3R5js --disable-tls --endpoint="minio-minio.apps.vlan611.rdu2.scalelab.redhat.com" --prefix=velero/kopia/perf-busy-data-cephrbd-20pods-2gb/

./kopia manifest list
010a01993b0934f019805f6ae347d426 213 2023-08-31 15:59:59 UTC type:maintenance
01dba7613b35412e1c49afe73bc305a0 1052 2023-08-31 16:02:10 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:93c0233c-f092-417f-989e-53e971679fe3 pod:busy-data-rbd-20pods-2gb-18-6df7459bdf-k7jbq pvc-uid:a52c0602-42fa-45f3-82fa-fd2a276137d0 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
6518781670c4c104c41c1b8b70060320 1052 2023-08-31 16:02:23 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:fbc97f52-78d2-4691-9d49-8835a8a8f893 pod:busy-data-rbd-20pods-2gb-19-79dfd95668-cttmp pvc-uid:c2660ef5-30aa-4e52-8ad4-8ab4c3fa636b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
dd63cbc26675f99379d75c1492bbff19 1052 2023-08-31 16:02:36 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:8bf8b043-f5e0-49c6-8d7e-7062ba4f8b2e pod:busy-data-rbd-20pods-2gb-2-7d6d6c55cd-cjx6x pvc-uid:14ef8302-bbc9-4b95-9302-17133bfe7ba9 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
aa969040a2cbb8548868c6c6122b2e67 1053 2023-08-31 16:02:49 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:ac1ccb7a-a8bc-4a36-9713-4695ba2573f4 pod:busy-data-rbd-20pods-2gb-20-6dd7cd8496-n7cnv pvc-uid:41c40d51-b020-443c-9bfa-e795c313858b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
9003be0534a02efe1c8dddb000d0a69a 1052 2023-08-31 16:03:02 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:a051dd4f-83bc-4b25-9324-9dc5b47b3463 pod:busy-data-rbd-20pods-2gb-3-584499c959-xqhkw pvc-uid:a39d31bf-c470-48db-a2bd-f43706673d0c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f28df4e791a8e4771a35adf7e28d8bc7 1052 2023-08-31 16:03:15 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:53257d44-11f5-4aac-829a-87751ea324b4 pod:busy-data-rbd-20pods-2gb-4-7c8b567d84-q5xtl pvc-uid:0c3cbdf6-7fcb-4b53-b5b2-1efc6d157b7c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f9ab5670efee51c8756586be98e7cb29 1053 2023-08-31 16:03:28 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:3b4714d8-38f1-4a11-b618-2e54841a411a pod:busy-data-rbd-20pods-2gb-5-7c7ccfbd5c-htmgf pvc-uid:880f924b-6e4b-4b54-8462-0cfc48d5c022 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
1563d949282f980558572bbf60f3d321 1052 2023-08-31 16:03:42 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:5f4a13c3-9f9a-4c57-95c5-e3aa661c09cc pod:busy-data-rbd-20pods-2gb-6-575f8db5fd-sb8c7 pvc-uid:5549f66d-553f-43a0-a644-8b3afb71f271 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
248925db2b78baa448b4ed2d79eedce7 1052 2023-08-31 16:03:55 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:ad81dec3-e988-461e-b3a1-f38c32c1ecc1 pod:busy-data-rbd-20pods-2gb-7-7f7468bc6c-kdtkt pvc-uid:9a64c98f-c53c-4d69-8248-634222da4a86 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
580a36d8da3b5d13ca57ac552cc86084 1052 2023-08-31 16:04:08 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:a0bd7276-b8ac-4572-82c8-618cd8dcd807 pod:busy-data-rbd-20pods-2gb-8-6c496c4776-6rd66 pvc-uid:58c80880-edf5-4293-bb50-948eccfe04f5 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
d12d773b30232d372803393fcf2735f0 1052 2023-08-31 16:04:22 UTC type:snapshot backup-uid:9ccb0660-eca4-4965-94a5-6f2b1d3a1d9a backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:9e92c684-419e-4380-a702-a20fc7ac0766 pod:busy-data-rbd-20pods-2gb-9-5bcf597b6f-5kmnr pvc-uid:9c27dce5-ccf8-46bf-bda5-f82eed6be03b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0

./kopia snapshot list --all
default@default:.
2023-08-31 16:01:58 UTC k3ea1140fcb740065896a3dc3c2a60dee 2.1 GB dgrwxrwxr-x files:6901 dirs:33
2023-08-31 16:02:11 UTC ka95783565de9957dc3c8896ead35d13b 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-10)
2023-08-31 16:02:24 UTC k018150994dabcdcaf3b8f130cfecf9ae 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-9)
2023-08-31 16:02:38 UTC k2ab7e0504761032f5338e7238026b7b5 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-8)
2023-08-31 16:02:51 UTC kb82dfaf32eb096eaba1cee7461fa743a 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-7)
2023-08-31 16:03:04 UTC k93ef7aef2b078a3c04e1ad019e9ef218 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-6)
2023-08-31 16:03:17 UTC k6c592f5070988d952f2a9a4c5fe734ea 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-5)
2023-08-31 16:03:30 UTC kc8e18529470db344ff2e4425dd5ad8f0 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-4)
2023-08-31 16:03:44 UTC k508f7ce9141d4d677c2daab03a34fb82 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-3)
2023-08-31 16:03:57 UTC k5415b55b9567f74a864372fb34c4d56e 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-2)
2023-08-31 16:04:10 UTC k11d498684c9ad382a000fe3c98b9110a 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

2nd namespace

kopia-backup-30pods-2gb Completed 0 0 2023-08-31 16:14:38 +0000 UTC 25d default
kopia-restore-30pods-2gb kopia-backup-30pods-2gb PartiallyFailed 2023-09-03 08:33:22 +0000 UTC 2023-09-03 08:36:23 +0000 UTC 19 10 2023-09-03 08:33:22 +0000 UTC

./kopia repository connect s3 --bucket="oadp-bucket" --access-key=Cvaz1RguhpAyuAYnAETQ --secret-access-key=+BSgIgkn0EvUT+kSeXcH8u4xo6MoHzSVRPG3R5js --disable-tls --endpoint="minio-minio.apps.vlan611.rdu2.scalelab.redhat.com" --prefix=velero/kopia/perf-busy-data-cephrbd-30pods-2gb

./kopia manifest list
561b1374fded9774751ebbd35cba46ac 213 2023-08-31 16:14:50 UTC type:maintenance
6f8a73acbbb71f272ed3382f9edc2ee4 1053 2023-08-31 16:19:18 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:3b506188-1c9a-44d8-b5da-03ae1619a212 pod:busy-data-rbd-30pods-2gb-27-5bcd8f6c4b-cjdmt pvc-uid:e864ce2d-5e01-4e0e-bf98-63567425cf80 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
a7b31b3194156dcc28315981832ffdd5 1053 2023-08-31 16:19:31 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:f01035df-b786-4f60-b708-ee370af84d50 pod:busy-data-rbd-30pods-2gb-28-86c4fd57cc-f78k4 pvc-uid:7b6bdc97-36a6-42dd-8fc7-6ff0f16cca5f snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
20ea9b490eb08fcc518b0ac545d26ba9 1053 2023-08-31 16:19:44 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:0cd4c0e6-01bf-4d29-953d-06dd21702f23 pod:busy-data-rbd-30pods-2gb-29-8677d44d5b-zp6mv pvc-uid:a3a3472e-ba1a-4deb-8143-a71c0071eb86 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
cbb880709a5b193ca3cea031741236fc 1051 2023-08-31 16:19:57 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:ea505fd4-a436-4055-b97f-6683248d63a2 pod:busy-data-rbd-30pods-2gb-3-668cf4fbcb-tzxk6 pvc-uid:c3ebb995-686b-4229-a25c-c08b59b4a1e9 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f02cadae1f3476b3b3d409941bc3f677 1052 2023-08-31 16:20:10 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:5cccb406-050f-415b-84c6-033664d82483 pod:busy-data-rbd-30pods-2gb-30-7f9896ccc4-m64jl pvc-uid:453045be-c5cb-4376-8bc4-22458865b32b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
813ec141a73037c5ca034b852137c0ea 1052 2023-08-31 16:20:23 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:ddf0ae7d-491c-4a00-9829-2fe04df55059 pod:busy-data-rbd-30pods-2gb-4-69cb876667-2tt7r pvc-uid:88bfac46-aaf2-491d-8348-cfbf7ee3ea0c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
11a477915f711172b42949ea0e098915 1051 2023-08-31 16:20:36 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:429746ee-bd0f-4410-8870-91e85088c9b8 pod:busy-data-rbd-30pods-2gb-5-675ffd8874-zg8s2 pvc-uid:fb183164-b7a8-46d7-a885-b7b38ef8f685 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
fc746cc61b5f0bad13dc324409188fd2 1051 2023-08-31 16:20:50 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:853e132d-2cad-4f55-a9a9-1ff3a3803fc0 pod:busy-data-rbd-30pods-2gb-6-59bb4576df-dkfbn pvc-uid:c69ee6fa-b9c1-40f2-8f38-16c4ef1e8199 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
35e793613e318d1f01b49ee2ab381aa8 1050 2023-08-31 16:21:03 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:a5be57c8-429a-40a4-8373-563490f14438 pod:busy-data-rbd-30pods-2gb-7-66fbb475f6-d2v5l pvc-uid:b08108a4-c3da-4bd8-8cd8-b8f3f494f37c snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
36132b36e8acbcc14afb1037a9c87583 1052 2023-08-31 16:21:16 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:cff0822b-43d7-4fd5-9d8f-82ffeb9e3b79 pod:busy-data-rbd-30pods-2gb-8-76f7df79b7-b5ghf pvc-uid:38905cc7-9abf-4317-a4e4-f8657d6de4df snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
251ee27b63faa7a5da6f4388a8aa399e 1052 2023-08-31 16:21:30 UTC type:snapshot backup-uid:06e957a5-a61f-4ea3-9555-3093d4a92b4f backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:23409050-0f5e-4a3d-ae81-3cb206775b55 pod:busy-data-rbd-30pods-2gb-9-7c9bb6fb6b-p2n58 pvc-uid:03f7a2a5-d27e-436e-aeef-740cfe3dd677 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0

./kopia snapshot list --all
default@default:.
2023-08-31 16:19:05 UTC k9ac51c064280785121f3bf9f16f83b20 2.1 GB dgrwxrwxr-x files:6901 dirs:33
2023-08-31 16:19:19 UTC k0cfbaf3d970dc089e1343b5d966caa72 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-10)
2023-08-31 16:19:32 UTC k81a153c76aa4d0f0ef818f81d40dcec9 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-9)
2023-08-31 16:19:45 UTC ka171e58e54b445c5085adeb5a93223af 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-8)
2023-08-31 16:19:58 UTC k0b53576a0d3341d7e0dcd3ac0d74c9cd 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-7)
2023-08-31 16:20:11 UTC k61661b05de336d17c1126f3d19edbbdf 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-6)
2023-08-31 16:20:25 UTC k2f96a92090290612aa607eda5744836e 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-5)
2023-08-31 16:20:38 UTC kddd5e1b5f917f3d4e91bebcdaf010a55 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-4)
2023-08-31 16:20:51 UTC k34e2903aa38bffd4ec360656183d0380 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-3)
2023-08-31 16:21:05 UTC k7760b4e0af268a5a89f82659151f296f 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-2)
2023-08-31 16:21:18 UTC k69019e0be52fe98444efc2673cd8245d 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

[duduvaa]

Bug verification failed.

Restore 12 out of 20 Pods

Errors:
Velero: pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot c9a58b3af12e8084f28122a13cc12872: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 8763611c0495c176efab5f24a8f798a2: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot c41015b930b8c5542c6764ea7654954e: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot f047de8ac56e25a26890d2a44f7a2d33: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 96fbac331b1484aab84f195d44544e7c: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 3ce11029ed4477f8f8ff4fe0b6377134: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot d459b46ebb68ef7f3d9e4e1f3112e512: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 5d8411cfb5254bc8a5f56b5fd9416847: snapshot not found

kopia-restore-20pods-2gb.tar.gz

[Lyndon-Li]

The problem is on backup side, so need to create new backups and the run the restore.

[duduvaa]

running backup&restore cycles - Restore still failed

image: Version: release-1.12-dev
last commit:
commit f72afc8 (HEAD -> main, origin/main, origin/HEAD)
Merge: a6d61ec 79b810e
Author: Xun Jiang/Bruce Jiang 59276555+blackpiglet@users.noreply.github.com
Date: Wed Sep 6 15:44:20 2023 +0800

Merge pull request #6760 from blackpiglet/6752_fix

Fix #6752: add namespace exclude check.

Velero: pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 299e8a649026d1cbb122cee437cea184: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 90447fc1d32b7ff712df6d620c970ab2: snapshot not found
pod volume restore failed: data path restore failed: Failed to run kopia restore: Unable to load snapshot 462058f6871d2e340ba80db3adb0d825: snapshot not found

backup plans status

NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
kopia-backup-20pods-2gb Completed 0 0 2023-09-06 16:20:02 +0000 UTC 29d default
kopia-backup-30pods-2gb Completed 0 0 2023-09-06 17:52:03 +0000 UTC 29d default

NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
kopia-restore-20pods-2gb kopia-backup-20pods-2gb PartiallyFailed 2023-09-06 21:16:21 +0000 UTC 2023-09-06 21:21:16 +0000 UTC 9 10 2023-09-06 21:16:21 +0000 UTC
kopia-restore-30pods-2gb kopia-backup-30pods-2gb PartiallyFailed 2023-09-06 21:31:29 +0000 UTC 2023-09-06 21:36:31 +0000 UTC 19 10 2023-09-06 21:31:28 +0000 UTC

kopia-restore-30pods-2gb.tar.gz
kopia-backup-30pods-2gb.tar.gz
kopia-restore-20pods-2gb.tar.gz
kopia-backup-20pods-2gb.tar.gz

[Lyndon-Li]

Looks like the number of failed PVRs has reduced significantly.
I didn't encounter this problem during my local verification, so I suspect it is related to the existing repo. We need to figure out what happened.

@duduvaa Could you run the same kopia manifest list and kopia snapshot list --all to the current repo and share the results?

[duduvaa]

20pods

kopia manifest list
1a56a3090e446ff3402a8a86af9843ea 213 2023-09-06 16:20:08 UTC type:maintenance
2bb6c4d269da10c805bb73a8a4e42d6f 1052 2023-09-06 16:22:22 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:d09993b6-37b4-4c83-9e6f-43705470fe7c pod:busy-data-rbd-20pods-2gb-18-6df7459bdf-527w4 pvc-uid:253a4f05-b857-4f1e-9837-aa6bbeb298af snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
63b2a9c679e3e12d347ec0d687e920b7 1053 2023-09-06 16:22:35 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:aeb29b52-344a-44c9-9f1b-b48997609acc pod:busy-data-rbd-20pods-2gb-19-79dfd95668-nxdl7 pvc-uid:bd0fe861-ea7d-4a45-b0e2-a8b6701f1ee6 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
9e4f84f0268e4bceb7a167306da83898 1052 2023-09-06 16:22:48 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:81f322e7-eca7-4a12-8255-facb661d01a0 pod:busy-data-rbd-20pods-2gb-2-7d6d6c55cd-qw26f pvc-uid:18279843-9011-4cfe-a473-6ecadca9c87b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
d641cc2773566f8959b8819c7682159c 1052 2023-09-06 16:23:00 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:75da4b0a-225b-4bbb-8a2f-23994b2bf0a2 pod:busy-data-rbd-20pods-2gb-20-6dd7cd8496-v4lx6 pvc-uid:5844f270-cd97-4e22-9e85-8da998536357 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
127c28a56a2059fd50ca9b1672a79df3 1052 2023-09-06 16:23:13 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:602b72bd-27a4-40c1-912a-1b38549bd699 pod:busy-data-rbd-20pods-2gb-3-584499c959-hxx6s pvc-uid:8510caf7-fafc-43ab-a268-8b5afdca5f79 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
900e973c547096f52067e1c21f93a4bf 1052 2023-09-06 16:23:25 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:0450354d-b9fc-4216-a0e0-86bbc10282b0 pod:busy-data-rbd-20pods-2gb-4-7c8b567d84-hvvd9 pvc-uid:ec2a19f8-1722-49e6-887a-c964e50a2b30 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
798b6c133035750a2c29ae2e1943a803 1051 2023-09-06 16:23:38 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:57cffc8d-e470-4819-b4ca-875f1c748da2 pod:busy-data-rbd-20pods-2gb-5-7c7ccfbd5c-cr2cs pvc-uid:3c95d0b1-ae14-473f-a0e8-13b5ce2cada7 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
63f1c9c8197810dd9533b7bf01e3b08c 1051 2023-09-06 16:23:51 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:56a29b4e-7fa3-422a-af88-b92c293cfc12 pod:busy-data-rbd-20pods-2gb-6-575f8db5fd-jnsnk pvc-uid:8faabcc6-7ff8-451e-b586-623ff0511d9e snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
eaea2ca2d364f97b81ff20c78f3e6ac8 1052 2023-09-06 16:24:03 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:854d09dd-2804-4824-a7ac-bd119f8b3e82 pod:busy-data-rbd-20pods-2gb-7-7f7468bc6c-62x2s pvc-uid:92ec0932-0b53-428d-b22c-8ebc14ffb9d9 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
f12b631f4026695268d75211c86f781e 1052 2023-09-06 16:24:16 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:1fcf77c2-ae1e-4f0a-b69e-c1a17238cb49 pod:busy-data-rbd-20pods-2gb-8-6c496c4776-psztt pvc-uid:20546744-c3e0-4aa8-bd39-bd794e43a706 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
01769ecc721f10eb1dff37e7e14f1f74 1052 2023-09-06 16:24:29 UTC type:snapshot backup-uid:6e448845-0afb-49bd-a4ba-1e43c5413b3c backup:kopia-backup-20pods-2gb hostname:default ns:perf-busy-data-cephrbd-20pods-2gb path:. pod-uid:87bc1fde-15f0-480a-9bdc-6d74e4d1c658 pod:busy-data-rbd-20pods-2gb-9-5bcf597b6f-mlgd9 pvc-uid:32831521-0217-46f4-8feb-a0532345ec51 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0

kopia snapshot list --all
default@default:.
2023-09-06 16:22:11 UTC kc5bcb31706fc822e76afc8b92d278a19 2.1 GB dgrwxrwxr-x files:6901 dirs:33
2023-09-06 16:22:24 UTC k4a2da40c28fb4997840c608a75db3646 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-10)
2023-09-06 16:22:36 UTC k8f41a5c771e18148b56747c753623dd0 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-9)
2023-09-06 16:22:49 UTC k63d4f177487293a2ec7db72410f4f939 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-8)
2023-09-06 16:23:02 UTC k923e22807563c9200350a2ad3b28cfb8 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-7)
2023-09-06 16:23:14 UTC k73eaf2187905d4cb3b246c7121fba138 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-6)
2023-09-06 16:23:27 UTC k07ab6c827ab1b44867635e988ea42fa2 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-5)
2023-09-06 16:23:39 UTC kbd094241be7a05bacfa2469f2620d477 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-4)
2023-09-06 16:23:52 UTC kd26d4b58591a89a337f6f91bf1210937 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-3)
2023-09-06 16:24:05 UTC k0731736e707b37001167c91b0bae9de9 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-2)
2023-09-06 16:24:17 UTC kdcc4357fabf183c55eaa3ffb65894fe5 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

30pods

kopia manifest list
22a012fa07ffd0da271be7f31ec041da 213 2023-09-06 17:52:09 UTC type:maintenance
6bd1fc5b3f30b27f813efe11811810c3 1052 2023-09-06 17:56:30 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:77afe3cb-9b5f-4b35-a005-a9011639ed09 pod:busy-data-rbd-30pods-2gb-27-5bcd8f6c4b-ljdmn pvc-uid:25919892-d62c-47a2-8ceb-a6fb54abe3ae snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
4535dcbcc4f16882b10d790bcd9d7eda 1053 2023-09-06 17:56:43 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:35aa5122-e23f-4c29-84dd-46740ab6d7a2 pod:busy-data-rbd-30pods-2gb-28-86c4fd57cc-8qj2l pvc-uid:6a57d9c9-cc4c-4775-a2c4-7a286b355b97 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
ab3ad8268511822428590d5f42489b31 1052 2023-09-06 17:56:55 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:a48d2330-3b07-4b03-aaf4-94c73310633d pod:busy-data-rbd-30pods-2gb-29-8677d44d5b-7t475 pvc-uid:57a175fa-0ac1-4dd9-a9f3-018e3d8940c0 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
b01551158d992a376a3cb2f777c09a16 1050 2023-09-06 17:57:08 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:bd64fcd8-c051-42ef-b663-e3dfce3a584c pod:busy-data-rbd-30pods-2gb-3-668cf4fbcb-jdcqp pvc-uid:db78c8c0-8c03-441d-bcfa-579383b37b04 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
014853d68bafbaf68578d89db295ff81 1051 2023-09-06 17:57:21 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:ad8c41c1-dd0b-405e-bdb4-0458c8b10721 pod:busy-data-rbd-30pods-2gb-30-7f9896ccc4-8cl2l pvc-uid:606e40e4-1149-4727-873d-34d3e92e7989 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
83e5a2433ec89cd3520a2f5dbe60acd3 1052 2023-09-06 17:57:34 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:be7db88c-0529-4d7a-9f01-61c44d99f15f pod:busy-data-rbd-30pods-2gb-4-69cb876667-7nvk6 pvc-uid:2aec3718-ee42-4b7b-9773-defaf023fff8 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
12085059217a606eb89fb295569ec273 1051 2023-09-06 17:57:46 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:1c7e4b9a-39bd-42c7-afd4-264ca1f4eb5a pod:busy-data-rbd-30pods-2gb-5-675ffd8874-tbjkv pvc-uid:b926037c-c815-46d7-a258-9922d57ed7e4 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
6482fcc05047336dd59d344ca83e0a80 1052 2023-09-06 17:57:59 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:c89221fe-49e2-4474-9fa7-a22f3da6c969 pod:busy-data-rbd-30pods-2gb-6-59bb4576df-9fbn4 pvc-uid:473a04e9-a0b3-46c3-ae48-ecc1f997482d snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
318f80d43de273803b95e751ee462431 1052 2023-09-06 17:58:12 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:d690cb0c-597e-4b9c-a8e4-55f20a5ae18e pod:busy-data-rbd-30pods-2gb-7-66fbb475f6-hxgzb pvc-uid:18f9583a-8c69-4182-b0dc-9f2d76b484ae snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
249cfcf960b6003c37cd2fe8e2b8e25b 1051 2023-09-06 17:58:24 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:45a151d6-ed8b-453e-b722-07ff1ac9dbf0 pod:busy-data-rbd-30pods-2gb-8-76f7df79b7-l2j98 pvc-uid:03ca69de-6ed8-4e06-82c2-063498bc4742 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0
938ad08454c7ee3161e7ad3dd6b7de27 1052 2023-09-06 17:58:37 UTC type:snapshot backup-uid:d4a0b1e2-ac6c-4ee0-a6dc-a1ae9e503a10 backup:kopia-backup-30pods-2gb hostname:default ns:perf-busy-data-cephrbd-30pods-2gb path:. pod-uid:a7a2e408-25df-440b-a7a3-2a9b905c7e17 pod:busy-data-rbd-30pods-2gb-9-7c9bb6fb6b-hlf4b pvc-uid:e84175db-5f74-4041-82b7-ad5657347ec4 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia username:default volume:vol-0

kopia snapshot list --all
default@default:.
2023-09-06 17:56:19 UTC k7a9841e9e51daf74392e602d5ef22709 2.1 GB dgrwxrwxr-x files:6901 dirs:33
2023-09-06 17:56:31 UTC k0bd54420aa6f12db5fb27e7892ec1f72 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-10)
2023-09-06 17:56:44 UTC kdfc132e7a5e300a65310e0bf43b45112 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-9)
2023-09-06 17:56:57 UTC k3ab4e05f8c7139fa7cf16c7131474860 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-8)
2023-09-06 17:57:10 UTC k082d6a088d9026339d3b8c2f8405fd6a 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-7)
2023-09-06 17:57:23 UTC k6a48ab73ecdb8df249cb4717e318a76b 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-6)
2023-09-06 17:57:35 UTC kcac74b8e25d9c2bfa9a26367cce28c18 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-5)
2023-09-06 17:57:48 UTC k0b56f4cd9751a6f81bd65bfa8bd70583 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-4)
2023-09-06 17:58:01 UTC ka6584ec02ef39711400b1632b5499751 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-3)
2023-09-06 17:58:13 UTC k106a5f847ee761a906cfb2c74da63c7d 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-2)
2023-09-06 17:58:26 UTC k5fbad3e6decf900e28b19115fe3ea13e 2.1 GB dgrwxrwxr-x files:6901 dirs:33 (latest-1,hourly-1,daily-1,weekly-1,monthly-1,annual-1)

[Lyndon-Li]

@duduvaa
It is weird that the snapshot path is still not correct, it seems the old image is still being used.
Have you changed the image for both velero server and node-agent?

[duduvaa]

Bug verified OK.
Tested namespaces with 20 & 30 pods. All pods are running. Restore 'Completed' OK.

@Lyndon-Li , thanks for the quick fix