[BACKPORT][v1.6.1][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones

[github-actions]

backport #7425

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: Be more careful while cleaning up stale replicas longhorn-manager#2432 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  It may be beneficial for users to increase the staleReplicaTimeout on their volumes to a large number (e.g. in days). This increase should have a similar effect except that it may cover even more situations. The downside is that this workaround has a time limit. If the user does not take corrective action before staleReplicaTimeout is exceeded, replicas will be deleted.

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: Add LastFailedAt and LastHealthyAt replica fields (backport #7715) #7804.
  The PR for the chart change is at: Add LastFailedAt and LastHealthyAt replica fields (backport #7715) #7804.

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Be more careful while cleaning up stale replicas (backport #2432) longhorn-manager#2546.

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at: Add skeleton for test_retain_potentially_useful_replicas_in_autosalvage_loop longhorn-tests#1699.
  The issue of automation test case implementation is at: [TEST][BUG] Longhorn may keep corrupted salvaged replicas and discard good ones #7803.

[roger-ryao]

Verified on v1.6.x-head 20240306

• longhorn v1.6.x-head 7e01ba1
• longhorn-manager v1.6.x-head longhorn/longhorn-manager@f542e20

The test steps

1. Be more careful while cleaning up stale replicas longhorn-manager#2432 (comment)

Result Passed

1. If you wait any length of time, ReplicaA is never cleaned up.
2. After deleting ReplicaB and waiting for the volume to rebuild from ReplicaA, the data is consistent.