[BACKPORT][v1.5.5][BUG] A replica may be incorrectly scheduled to a node with an existing failed replica

[github-actions]

backport #8043

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps are at: [BUG] A replica may be incorrectly scheduled to a node with an existing failed replica #8043 (comment).
  The test steps are at: Consider a node with a failed reusable replica as still used longhorn-manager#2650 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at: [BUG] A replica may be incorrectly scheduled to a node with an existing failed replica #8043 (comment).

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Consider a node with a failed reusable replica as still used (backport #2650) longhorn-manager#2679.

• Which areas/issues this PR might have potential impacts on?
  In some failed-replica situations, scheduling behavior will be slightly changed. Longhorn will not schedule a new replica to a node that already contains a failed replica until the failed replica becomes unusable or replica-replenishment-wait-interval is exceeded.

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  We have pretty solid unit/integration testing within the longhorn-manager PR. I don't recommend a new end-to-end test at this time.
  https://github.com/longhorn/longhorn-manager/pull/2650/files#diff-6bfd30bb63aa743911a9b4f7f2cf9e3743f133a2dbd0e7ebfee6293c9bcd7541

[chriscchien]

Verified pass on longhorn v1.5.x (longhorn-manager 142c85)

Following test steps, did not encounter problem and there is no extra replica scheduled on the only one node in the cluster.