[BUG] Backup related test cases failed

[yangchiu]

Describe the bug

Lots of backup related test cases fail on master-head (longhorn-manager 50acc813):

test_backing_image.test_backup_with_backing_image[nfs]
test_basic.test_restore_basic[s3]
test_ha_backup_deletion_recovery[nfs]
test_backing_image.test_csi_backup_with_backing_image[nfs]
test_basic.test_dr_volume_with_all_backup_blocks_deleted[nfs]
test_basic.test_backup[nfs]

just to name a few.

Create this ticket first and investigate them one by one to see if there is anything missing or the test code should be refined.

To Reproduce

Expected behavior

Support bundle for troubleshooting

Environment

• Longhorn version: master-head
• Impacted volume (PV):
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: v1.28.4+k3s1
  • Number of control plane nodes in the cluster:
  • Number of worker nodes in the cluster:
• Node config
  • OS type and version: sles 15-sp5
  • Kernel version:
  • CPU per node:
  • Memory per node:
  • Disk type (e.g. SSD/NVMe/HDD):
  • Network bandwidth between the nodes (Gbps):
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

[shuo-wu]

There is a regression caused by longhorn/longhorn-manager#2611 and the fix is in longhorn/longhorn-manager#2621

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at controller: Fix issue that regular restore volume won't be detached longhorn-manager#2621

• Which areas/issues this PR might have potential impacts on?
  Area Restore volume
  Issues

[yangchiu]

Verified passed on master-head (longhorn-manager 43013ae1) and v1.5.x-head (longhorn-manager 079922fa). Failed backup related test cases passed now.

Test results:
v1.5.x-head: https://ci.longhorn.io/job/public/job/v1.5.x/job/v1.5.x-longhorn-tests-sles-amd64/308/
master-head: https://ci.longhorn.io/job/public/job/master/job/sles/job/amd64/job/longhorn-tests-sles-amd64/828/