[Remote Store] Add remote segment transfer stats on NodesStats API #9168
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
|
Compatibility status: |
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #9168 +/- ##
============================================
+ Coverage 71.03% 71.07% +0.04%
- Complexity 57318 57404 +86
============================================
Files 4770 4771 +1
Lines 270540 270623 +83
Branches 39561 39561
============================================
+ Hits 192186 192356 +170
+ Misses 62147 62081 -66
+ Partials 16207 16186 -21
|
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
shourya035
changed the title
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Gradle Check (Jenkins) Run Completed with:
|
sachinpkale
reviewed
Aug 10, 2023
server/src/main/java/org/opensearch/index/remote/RemoteSegmentStats.java
Outdated
Show resolved
Hide resolved
server/src/internalClusterTest/java/org/opensearch/indices/stats/IndexStatsIT.java
Show resolved
Hide resolved
.../internalClusterTest/java/org/opensearch/remotestore/RemoteSegmentStatsFromNodesStatsIT.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>
Gradle Check (Jenkins) Run Completed with:
|
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
|
Overall changes LGTM. |
|
Build is failing due to flaky test: #9116 |
Gradle Check (Jenkins) Run Completed with:
|
sachinpkale
approved these changes
Aug 14, 2023
6 tasks
shourya035
added a commit
to shourya035/OpenSearch
that referenced
this pull request
Aug 14, 2023
…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> (cherry picked from commit c9dbd90)
ashking94
reviewed
Aug 16, 2023
| @@ -1377,6 +1384,12 @@ public MergeStats mergeStats() { | |||
| public SegmentsStats segmentStats(boolean includeSegmentFileSizes, boolean includeUnloadedSegments) { | |||
| SegmentsStats segmentsStats = getEngine().segmentsStats(includeSegmentFileSizes, includeUnloadedSegments); | |||
| segmentsStats.addBitsetMemoryInBytes(shardBitsetFilterCache.getMemorySizeInBytes()); | |||
| // Populate remote_store stats only if the index is remote store backed | |||
There was a problem hiding this comment.
We are doing an add here and one explicit add in NoOpEngine.java class -
This seem to be adding same stats twice.
ashking94
reviewed
Aug 16, 2023
| downloadBytesFailed = in.readLong(); | ||
| downloadBytesSucceeded = in.readLong(); | ||
| maxRefreshTimeLag = in.readLong(); | ||
| maxRefreshBytesLag = in.readLong(); |
There was a problem hiding this comment.
can this be cumulative refresh bytes lag itself?
ashking94
reviewed
Aug 16, 2023
shourya035
added a commit
to shourya035/OpenSearch
that referenced
this pull request
Aug 16, 2023
…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> (cherry picked from commit c9dbd90)
6 tasks
4 tasks
31 tasks
kaushalmahi12
pushed a commit
to kaushalmahi12/OpenSearch
that referenced
this pull request
Sep 12, 2023
…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic
pushed a commit
to brusic/OpenSearch
that referenced
this pull request
Sep 25, 2023
…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adding remote segment transfer stats on the NodesStats API output. As of now, we are including the following metrics:
total_uploads.started_bytes: Amount of segment payload attempted to be uploaded to the remote storetotal_uploads.succeeded_bytes: Amount of segment payload successfully uploaded to the remote storetotal_uploads.failed_bytes: Amount of segment payload failed to be uploaded to the remote storemax_refresh_time_lag_in_millis: Max refresh lag in milliseconds across all the shards on the nodemax_refresh_size_lag_in_bytes: Max refresh lag in size across all the shards on the nodetotal_downloads.started_bytes: Amount of segment payload attempted to be downloaded from the remote storetotal_downloads.succeeded_bytes: Amount of segment payload successfully downloaded from the remote storetotal_downloads.failed_bytes: Amount of segment payload failed to be downloaded from the remote storeThese are added to the
indices.segmentsection of the NodesStats API outputAdded integ test cases to ensure that:
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.