[Backport 2.x] Decouple replication lag from logic to fail stale replicas (#9507)

[mch2]

Manual backport of #9507

[github-actions]

Compatibility status:

Checks if related components are compatible with change bee6497

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24163/
• CommitID: bee6497
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24170/
• CommitID: bee6497
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24178/
• CommitID: bee6497
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/24196/
• CommitID: bee6497
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/24246/
• CommitID: bee6497

[codecov]

Codecov Report

Merging #9705 (bee6497) into 2.x (9e41df4) will decrease coverage by 0.03%.
Report is 6 commits behind head on 2.x.
The diff coverage is 75.66%.

@@             Coverage Diff              @@
##                2.x    #9705      +/-   ##
============================================
- Coverage     70.86%   70.83%   -0.03%     
- Complexity    57786    57839      +53     
============================================
  Files          4776     4788      +12     
  Lines        273121   273359     +238     
  Branches      40198    40220      +22     
============================================
+ Hits         193537   193644     +107     
- Misses        63032    63168     +136     
+ Partials      16552    16547       -5     

Files Changed	Coverage Δ
...java/org/opensearch/common/crypto/DataKeyPair.java	0.00% <0.00%> (ø)
...ain/java/org/opensearch/common/unit/TimeValue.java	87.50% <ø> (+1.25%)	⬆️
...ibs/core/src/main/java/org/opensearch/Version.java	82.45% <ø> (-1.32%)	⬇️
.../src/main/java/org/opensearch/core/ParseField.java	95.65% <ø> (ø)
...ava/org/opensearch/core/action/ActionListener.java	78.50% <0.00%> (-1.87%)	⬇️
...g/opensearch/core/common/bytes/BytesReference.java	88.57% <ø> (+14.28%)	⬆️
.../core/common/io/stream/NamedWriteableRegistry.java	100.00% <ø> (ø)
.../opensearch/core/common/io/stream/StreamInput.java	88.45% <ø> (-0.40%)	⬇️
...opensearch/core/common/io/stream/StreamOutput.java	94.88% <ø> (-0.81%)	⬇️
.../opensearch/core/common/settings/SecureString.java	78.78% <ø> (ø)
... and 57 more

... and 455 files with indirect coverage changes