[BUG] Create volume(v1) faulted

[chriscchien]

Describe the bug (🐛 if you encounter this issue)

Create v1 volume faulted(UI and dynamic provision) in a fresh installed Longhorn cluster and test_volume_basic faild with latest images

longhorn-manager log

2024-01-04T03:12:15.707422491Z time="2024-01-04T03:12:15Z" level=info msg="Request (user: system:serviceaccount:longhorn-system:longhorn-service-account, longhorn.io/v1beta2, Kind=Engine, namespace: longhorn-system, name: test-vol-e-0, operation: CREATE) patchOps: [{\"op\": \"replace\", \"path\": \"/metadata/labels\", \"value\": {\"longhornnode\":\"\",\"longhornvolume\":\"test-vol\"}},{\"op\": \"replace\", \"path\": \"/metadata/finalizers\", \"value\": [\"longhorn.io\"]}]" func="admission.(*Handler).admit" file="admission.go:115" service=admissionWebhook
2024-01-04T03:12:15.715818686Z W0104 03:12:15.715674       1 warnings.go:70] unknown field "spec.snapshotMaxCount"
2024-01-04T03:12:15.715831513Z W0104 03:12:15.715690       1 warnings.go:70] unknown field "spec.snapshotMaxSize"
2024-01-04T03:12:15.715836795Z W0104 03:12:15.715695       1 warnings.go:70] unknown field "status.snapshotMaxCount"
2024-01-04T03:12:15.715841470Z W0104 03:12:15.715700       1 warnings.go:70] unknown field "status.snapshotMaxSize"
2024-01-04T03:12:15.739987767Z W0104 03:12:15.739648       1 warnings.go:70] unknown field "spec.snapshotMaxCount"
2024-01-04T03:12:15.740006548Z W0104 03:12:15.739669       1 warnings.go:70] unknown field "spec.snapshotMaxSize"
2024-01-04T03:12:15.740011094Z W0104 03:12:15.739673       1 warnings.go:70] unknown field "status.snapshotMaxCount"
2024-01-04T03:12:15.740016160Z W0104 03:12:15.739678       1 warnings.go:70] unknown field "status.snapshotMaxSize"
2024-01-04T03:12:15.816575171Z time="2024-01-04T03:12:15Z" level=info msg="Cannot find a reusable failed replicas for volume test-vol" func="scheduler.(*ReplicaScheduler).CheckAndReuseFailedReplica" file="replica_scheduler.go:542"

To Reproduce

1. Deploy Longhorn v1.6.0-dev
2. Create volume and attach to a node
3. Volume faulted

Expected behavior

Volume healthy

Support bundle for troubleshooting

supportbundle_b53994bd-dcd4-41cb-95f7-ad2080768de7_2024-01-04T03-15-18Z.zip

Environment

• Longhorn version: v1.6.0-dev (longhorn-manager 2e69c0)
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl): kubectl
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: v1.28.2+k3s1

Additional context

N/A

[c3y1huang]

Due to #6563 (comment).

[innobead]

#6634 merged.

[innobead]

Assigned to @FrankYang0529 . Please follow up with @chriscchien testing.

[chriscchien]

Deploy longhorn( longhorhn dd7163) again, volume operation worked well and and rerun test_volume_basic on pipeline passed, close this ticket as the issue did not happen anymore, thank you.