[Remote Store] Add tracker factory to manage remote store stats trackers #9546

BhumikaSaini-Amazon · 2023-08-25T10:54:27Z

Description

Currently, RemoteSegmentTransferTracker is strongly coupled with RemoteStorePressureService. RemoteStorePressureService is the provider of the trackers while it should just be consuming it. This PR decouples RemoteSegmentTransferTracker from RemoteStorePressureService by introducing the concept of a factory that manages the stats trackers.
Currently, RemoteSegmentTransferTracker allows setting a different moving average window size for upload bytes, upload speed, and upload time. These don't add much value and would rather deteriorate the user experience. We change these to apply a single window size for moving average stat calculations.

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-25T11:32:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23474/
CommitID: acf2041
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-25T11:33:13Z

Compatibility status:

Checks if related components are compatible with change acf2041

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-25T12:09:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23479/
CommitID: 7685bda
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-25T12:26:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23478/
CommitID: 0335986
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-25T12:31:39Z

Compatibility status:

Checks if related components are compatible with change 188c7e2

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

server/src/main/java/org/opensearch/index/remote/RemoteSegmentTransferTracker.java

server/src/main/java/org/opensearch/index/remote/RemoteStorePressureSettings.java

server/src/main/java/org/opensearch/index/shard/IndexShard.java

server/src/main/java/org/opensearch/index/remote/RemoteStorePressureService.java

server/src/main/java/org/opensearch/indices/cluster/IndicesClusterStateService.java

github-actions · 2023-08-25T14:06:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: TIMEOUT ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23481/
CommitID: 188c7e2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

...a/org/opensearch/action/admin/cluster/remotestore/stats/TransportRemoteStoreStatsAction.java

.../opensearch/action/admin/cluster/remotestore/stats/TransportRemoteStoreStatsActionTests.java

server/src/test/java/org/opensearch/index/remote/RemoteSegmentTransferTrackerTests.java

server/src/test/java/org/opensearch/index/remote/RemoteStorePressureServiceTests.java

server/src/test/java/org/opensearch/index/remote/RemoteStoreStatsTrackerFactoryTests.java

ashking94

Have requested couple of changes, please take a look.

ashking94

the diff (since last approval) looks good.

github-actions · 2023-08-29T12:56:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23712/
CommitID: e2c6379
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-29T17:54:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23738/
CommitID: 8b28c62
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

BhumikaSaini-Amazon · 2023-08-29T18:00:32Z

org.opensearch.index.shard.RemoteIndexShardTests.testNRTReplicaWithRemoteStorePromotedAsPrimaryRefreshCommit has been reported as flaky => #9598

github-actions · 2023-08-29T18:30:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23741/
CommitID: a364524
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

…egmentTransferTrackerTests Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

…tory Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

…eSettings Signed-off-by: Bhumika Saini <sabhumik@amazon.com>

github-actions · 2023-08-31T04:44:20Z

Compatibility status:

Checks if related components are compatible with change cc3884c

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-08-31T04:55:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23908/
CommitID: cc3884c

…ers (#9546) --------- Signed-off-by: Bhumika Saini <sabhumik@amazon.com> (cherry picked from commit c294c91) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ers (#9546) (#9654) --------- (cherry picked from commit c294c91) Signed-off-by: Bhumika Saini <sabhumik@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ers (opensearch-project#9546) --------- Signed-off-by: Bhumika Saini <sabhumik@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…ers (opensearch-project#9546) --------- Signed-off-by: Bhumika Saini <sabhumik@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…ers (opensearch-project#9546) --------- Signed-off-by: Bhumika Saini <sabhumik@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

BhumikaSaini-Amazon force-pushed the tracker-factory branch from 0335986 to 7685bda Compare August 25, 2023 11:59

sachinpkale reviewed Aug 25, 2023

View reviewed changes