Fix local segments stats update in RemoteStoreRefreshListener

[ashking94]

Description

Today, if a refresh occurs for an IndexShard, the afterRefresh method of ReferenceManager.RefreshListener (Lucene class) objects are invoked. RemoteStoreRefreshListener, currently, has the responsibility of updating the segment tracker with the upto date information of segment files which are part of the most recent refresh. It also has the responsibility of uploading the segments to the remote store after each refresh and then updating the segment tracker with the relevant information about the upload. This information is exposed via remote store stats api and is also used for rejection incoming write requests when there is a lag between the local store and the remote store (remote segment backpressure). While updating the tracker with local state happens at the beginning of the performAfterRefresh, there can be cases where the retry could have been blocking the performAfterRefresh (& hence the local state update in segment tracker). This would lead to incorrect information being saved in tracker due to the tight coupling.

With this PR, we aim to introduce a new method inside the CloseableRetryableRefreshListener which gets executed exactly once. The idempotent logic can continue to reside in the performAfterRefreshWithPermit method. We are also adding tests for existing classes that covers more flows.

Related Issues

Resolves #8717

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20439/
• CommitID: f0da81f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/20445/
• CommitID: 11493a5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicaHasDiffFilesThanPrimary
      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.remotestore.multipart.RemoteStoreMultipartIT.testStaleCommitDeletionWithInvokeFlush
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testPitCreatedOnReplica
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testNodeDropWithOngoingReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/20476/
• CommitID: a80c8e5
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8758 (ba0f515) into main (bb78930) will decrease coverage by 0.04%.
Report is 1 commits behind head on main.
The diff coverage is 74.30%.

@@             Coverage Diff              @@
##               main    #8758      +/-   ##
============================================
- Coverage     71.06%   71.02%   -0.04%     
- Complexity    57314    57331      +17     
============================================
  Files          4766     4765       -1     
  Lines        270453   270478      +25     
  Branches      39555    39552       -3     
============================================
- Hits         192197   192115      -82     
- Misses        62045    62199     +154     
+ Partials      16211    16164      -47     

Files Changed	Coverage Δ
.../opensearch/client/indices/CreateIndexRequest.java	75.55% <0.00%> (-1.12%)	⬇️
...opensearch/client/slm/SnapshotLifecyclePolicy.java	0.00% <ø> (ø)
...ch/client/slm/SnapshotLifecyclePolicyMetadata.java	0.00% <ø> (ø)
.../opensearch/client/slm/SnapshotLifecycleStats.java	0.00% <ø> (ø)
...rch/client/slm/SnapshotRetentionConfiguration.java	0.00% <ø> (ø)
...h/script/mustache/MultiSearchTemplateResponse.java	59.01% <ø> (ø)
...rg/opensearch/index/rankeval/RankEvalResponse.java	88.57% <ø> (-1.43%)	⬇️
...va/org/opensearch/index/rankeval/RankEvalSpec.java	97.36% <ø> (ø)
...a/org/opensearch/index/rankeval/RatedDocument.java	81.03% <0.00%> (ø)
...va/org/opensearch/index/rankeval/RatedRequest.java	91.79% <ø> (ø)
... and 133 more

... and 450 files with indirect coverage changes

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.cluster.routing.allocation.decider.DiskThresholdDeciderIT.testIndexCreateBlockIsRemovedWhenAnyNodesNotExceedHighWatermarkWithAutoReleaseEnabled

• URL: https://build.ci.opensearch.org/job/gradle-check/20618/
• CommitID: 3910aab
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[ashking94]

Rebased with main and committed as single commit as it was getting difficult to rebase every time.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/21878/
• CommitID: 56ba920

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21880/
• CommitID: 5dfaa9c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[ashking94]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21880/
• CommitID: 5dfaa9c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

Flaky tests - #9048, #9130, #8887, #9048

[opensearch-trigger-bot]

Compatibility status:

> Task :checkCompatibility
Checking compatibility for: https://github.com/opensearch-project/reporting.git with ref: main
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git]
Compatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 22m 57s

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21942/
• CommitID: 382e0ad
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot]

Compatibility status:

> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 23m 14s

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/21951/
• CommitID: ba0f515