[BACKPORT][v1.6.1][BUG] A replica may be incorrectly scheduled to a node with an existing failed replica

[github-actions]

backport #8043

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps are at: [BUG] A replica may be incorrectly scheduled to a node with an existing failed replica #8043 (comment).
  The test steps are at: Consider a node with a failed reusable replica as still used longhorn-manager#2650 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at: [BUG] A replica may be incorrectly scheduled to a node with an existing failed replica #8043 (comment).

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Consider a node with a failed reusable replica as still used (backport #2650) longhorn-manager#2677.

• Which areas/issues this PR might have potential impacts on?
  In some failed-replica situations, scheduling behavior will be slightly changed. Longhorn will not schedule a new replica to a node that already contains a failed replica until the failed replica becomes unusable or replica-replenishment-wait-interval is exceeded.

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  We have pretty solid unit/integration testing within the longhorn-manager PR. I don't recommend a new end-to-end test at this time.
  https://github.com/longhorn/longhorn-manager/pull/2650/files#diff-6bfd30bb63aa743911a9b4f7f2cf9e3743f133a2dbd0e7ebfee6293c9bcd7541

[chriscchien]

I can reproduce the issue on longhorn-manager v1.6.x 195495 with below information, and looks like it is because the backport PR longhorn/longhorn-manager#2677 not merged yet, I will test agagin after PR merged, thank you.

> k get replicas -A
NAMESPACE         NAME                                                  DATA ENGINE   STATE     NODE             DISK                                   INSTANCEMANAGER                                     IMAGE                                    AGE
longhorn-system   pvc-3d5e675d-b13a-4fe6-991b-9c3650eb98a7-r-dd1a2cfc   v1            stopped   ip-172-31-36-1   2dc222d3-7afa-4090-b8bf-8cf8e9dcb302                                                                                                13m
longhorn-system   pvc-3d5e675d-b13a-4fe6-991b-9c3650eb98a7-r-a8f5a10e   v1            stopped                                                                                                                                                        77s
longhorn-system   pvc-3d5e675d-b13a-4fe6-991b-9c3650eb98a7-r-1481e144   v1            stopped                                                                                                                                                        77s
longhorn-system   pvc-3d5e675d-b13a-4fe6-991b-9c3650eb98a7-r-293708ce   v1            running   ip-172-31-36-1   2dc222d3-7afa-4090-b8bf-8cf8e9dcb302   instance-manager-751d319e09110500b467860c15517312   longhornio/longhorn-engine:v1.6.x-head   13m

supportbundle_0c0f1074-d169-4eaa-9a7a-8d2b0d40497a_2024-03-08T03-47-08Z.zip

[chriscchien]

Verified pass on longhorn v1.6.x (longhorn-manager 36ad9f)

Following test stpes, did not encounter problem and there is no extra replica scheduled on the only one node in the cluster.