[BUG] v2 volume becomes faulted and detached after deleting one replica during full restoration

[derekbit]

Describe the bug

v2 volume becomes faulted and detached after deleting one replica during full restoration.

To Reproduce

1. Create a v2 volume with 3 replicas
2. Create a backup
3. Restore a v2 volume from the backup
4. Delete one replica during restoration.
5. The volume becomes faulted and detached after restoration.

Expected behavior

Support bundle for troubleshooting

Environment

• Longhorn version:
• Impacted volume (PV):
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of control plane nodes in the cluster:
  • Number of worker nodes in the cluster:
• Node config
  • OS type and version:
  • Kernel version:
  • CPU per node:
  • Memory per node:
  • Disk type (e.g. SSD/NVMe/HDD):
  • Network bandwidth between the nodes (Gbps):
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

<!-Please add any other context about the problem here.-->

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

After the fix, the behavior is

• The restoration should succeed
• Offline rebuilding is triggered right after the finish of the volume restoration
• The volume becomes detached and not faulted in the end

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

longhorn/longhorn-manager#2439

• Which areas/issues this PR might have potential impacts on?
  Area: v2 volume, restoration
  Issues

[chriscchien]

Verified pass on longhorn master(longhorn-manager 2ac44e) with test stpes

Delete replica when v2 volume is restoring, the volume will finish restore first and then start offline rebuilding, volume become detached(ready for workload) in the end, the data intact.