[Remote Store] Add cumulative bytes lag to NodesStats #9393

shourya035 · 2023-08-16T15:20:49Z

Description

Adding cumulative bytes lag to the remote_store section of NodesStats. The new API output looks like the following:

"remote_store": {
    "upload": {
        "total_uploads": {
            "started": "4.4mb",
            "started_bytes": 4708635,
            "succeeded": "4.4mb",
            "succeeded_bytes": 4708635,
            "failed": "0b",
            "failed_bytes": 0
        },
        "refresh_size_lag": {
            "total": "0b",
            "total_bytes": 0,
            "max": "0b",
            "max_bytes": 0
        },
        "max_refresh_time_lag": "0s",
        "max_refresh_time_lag_in_millis": 0
    }

Related Issues

N/A

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

opensearch-trigger-bot · 2023-08-16T15:57:46Z

Compatibility status:

Checks if related components are compatible with change 8d153ab

Incompatible components

Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-16T16:03:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22902/
CommitID: 3d5d827

codecov · 2023-08-16T16:05:14Z

Codecov Report

Merging #9393 (313de19) into main (8d153ab) will decrease coverage by 0.08%.
Report is 12 commits behind head on main.
The diff coverage is 90.90%.

@@             Coverage Diff              @@
##               main    #9393      +/-   ##
============================================
- Coverage     71.17%   71.10%   -0.08%     
+ Complexity    57480    57470      -10     
============================================
  Files          4777     4777              
  Lines        270821   270837      +16     
  Branches      39586    39588       +2     
============================================
- Hits         192755   192572     -183     
- Misses        61868    62089     +221     
+ Partials      16198    16176      -22

Files Changed	Coverage Δ
...rg/opensearch/index/remote/RemoteSegmentStats.java	`96.00% <90.90%> (-1.62%)`	⬇️

... and 457 files with indirect coverage changes

github-actions · 2023-08-18T07:49:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23015/
CommitID: 0949570
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

.../internalClusterTest/java/org/opensearch/remotestore/RemoteSegmentStatsFromNodesStatsIT.java

ashking94

lgtm

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2023-08-18T09:49:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23020/
CommitID: db627b1

github-actions · 2023-08-18T09:56:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23021/
CommitID: 93eb966

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

github-actions · 2023-08-18T10:24:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23022/
CommitID: d9f2327
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

opensearch-trigger-bot · 2023-08-18T12:08:16Z

Compatibility status:

Checks if related components are compatible with change 8e95a82

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

github-actions · 2023-08-18T12:09:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23025/
CommitID: 313de19

ashking94

lgtm

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> (cherry picked from commit 784a473)

opensearch-trigger-bot · 2023-08-25T07:59:07Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9393-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 784a473b3d3891e41afca3588dd3eda7a7bd3def
# Push it to GitHub
git push --set-upstream origin backport/backport-9393-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9393-to-2.x.

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Kiran Reddy <kkreddy@amazon.com>

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

shourya035 added 2 commits August 16, 2023 16:40

Changing bytes lag statistic to sum instead of max

70590e6

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Merge branch 'opensearch-project:main' into stats-rollup-max-bytes-lag

3d5d827

shourya035 changed the title ~~[Remote Store] Stats rollup max bytes lag~~ [Remote Store] Change max bytes lag to cumulative bytes lag Aug 17, 2023

shourya035 marked this pull request as ready for review August 17, 2023 05:22

ashking94 reviewed Aug 18, 2023

View reviewed changes

.../internalClusterTest/java/org/opensearch/remotestore/RemoteSegmentStatsFromNodesStatsIT.java Show resolved Hide resolved

ashking94 suggested changes Aug 18, 2023

View reviewed changes

shourya035 added 2 commits August 18, 2023 14:37

Renaming bytes lag fields and adding stream version checks

db627b1

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Adding new PR link to CHANGELOG

93eb966

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Retrigger Integs

d9f2327

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Retrigger Integs

313de19

Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

sachinpkale approved these changes Aug 21, 2023

View reviewed changes

sachinpkale removed the skip-changelog label Aug 21, 2023

ashking94 approved these changes Aug 21, 2023

View reviewed changes

sachinpkale merged commit 784a473 into opensearch-project:main Aug 21, 2023
12 checks passed

shourya035 deleted the stats-rollup-max-bytes-lag branch August 21, 2023 05:30

austintlee pushed a commit to austintlee/OpenSearch that referenced this pull request Aug 25, 2023

[Remote Store] Add cumulative bytes lag to NodesStats (opensearch-pro…

4403beb

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

shourya035 mentioned this pull request Aug 25, 2023

[Backport-2.x] Adding extra fields under RemoteStore stats in NodeStats #9544

Merged

6 tasks

ashking94 added the backport 2.x Backport to 2.x branch label Aug 25, 2023

opensearch-trigger-bot bot added the backport-failed label Aug 25, 2023

Gaganjuneja pushed a commit to Gaganjuneja/OpenSearch that referenced this pull request Aug 28, 2023

[Remote Store] Add cumulative bytes lag to NodesStats (opensearch-pro…

7804176

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

sachinpkale mentioned this pull request Sep 6, 2023

[Meta] Remote Store: 2.10.0 - Release Tracking #9805

Closed

31 tasks

kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023

[Remote Store] Add cumulative bytes lag to NodesStats (opensearch-pro…

61f7cf0

…ject#9393) --------- Signed-off-by: Shourya Dutta Biswas <114977491+shourya035@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add cumulative bytes lag to NodesStats #9393

[Remote Store] Add cumulative bytes lag to NodesStats #9393

shourya035 commented Aug 16, 2023 •

edited

opensearch-trigger-bot bot commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

codecov bot commented Aug 16, 2023 •

edited

github-actions bot commented Aug 18, 2023

ashking94 left a comment

github-actions bot commented Aug 18, 2023

github-actions bot commented Aug 18, 2023

github-actions bot commented Aug 18, 2023

opensearch-trigger-bot bot commented Aug 18, 2023

github-actions bot commented Aug 18, 2023

ashking94 left a comment

opensearch-trigger-bot bot commented Aug 25, 2023

[Remote Store] Add cumulative bytes lag to NodesStats #9393

[Remote Store] Add cumulative bytes lag to NodesStats #9393

Conversation

shourya035 commented Aug 16, 2023 • edited

Description

Related Issues

Check List

opensearch-trigger-bot bot commented Aug 16, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 16, 2023 • edited

Codecov Report

github-actions bot commented Aug 18, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 18, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 18, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 18, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Aug 25, 2023

shourya035 commented Aug 16, 2023 •

edited

codecov bot commented Aug 16, 2023 •

edited