Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store #7394

Merged
merged 9 commits into from May 15, 2023

Conversation

ankitkala
Copy link
Member

Description

This change adds a hook to publish the notification to replicas when a segments are uploaded to the remote store.
This change also gates this behind a new feature flag for Segment Replication with remote store..

Related Issues

#5578

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Contributor

github-actions bot commented May 3, 2023

Gradle Check (Jenkins) Run Completed with:

@ankitkala ankitkala force-pushed the remote_upload_hook branch 2 times, most recently from 1067906 to 511b605 Compare May 11, 2023 12:14
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link

codecov-commenter commented May 11, 2023

Codecov Report

Merging #7394 (e381c2c) into main (e44b3f1) will increase coverage by 0.09%.
The diff coverage is 71.42%.

@@             Coverage Diff              @@
##               main    #7394      +/-   ##
============================================
+ Coverage     70.47%   70.57%   +0.09%     
+ Complexity    59728    59726       -2     
============================================
  Files          4896     4896              
  Lines        286798   286807       +9     
  Branches      41331    41335       +4     
============================================
+ Hits         202133   202417     +284     
+ Misses        68078    67724     -354     
- Partials      16587    16666      +79     
Impacted Files Coverage Δ
.../main/java/org/opensearch/index/IndexSettings.java 84.19% <0.00%> (-0.44%) ⬇️
...in/java/org/opensearch/index/shard/IndexShard.java 70.64% <66.66%> (+0.58%) ⬆️
...nsearch/index/shard/CheckpointRefreshListener.java 91.66% <75.00%> (-8.34%) ⬇️
...search/index/shard/RemoteStoreRefreshListener.java 81.56% <100.00%> (-0.33%) ⬇️

... and 465 files with indirect coverage changes

…after segment upload to remote store

Signed-off-by: Ankit Kala <ankikala@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <ankikala@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <ankikala@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Ankit Kala <ankikala@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure

Signed-off-by: Ankit Kala <ankikala@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@mch2 mch2 merged commit 6f65afb into opensearch-project:main May 15, 2023
10 checks passed
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request May 23, 2023
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Added changelog

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix the test

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* gradle check

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed PR comments

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed comments 3

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix Changelog entry

Signed-off-by: Ankit Kala <ankikala@amazon.com>

---------

Signed-off-by: Ankit Kala <ankikala@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
ashking94 pushed a commit to ashking94/OpenSearch that referenced this pull request May 24, 2023
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Added changelog

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix the test

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* gradle check

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed PR comments

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed comments 3

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix Changelog entry

Signed-off-by: Ankit Kala <ankikala@amazon.com>

---------

Signed-off-by: Ankit Kala <ankikala@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
gbbafna pushed a commit that referenced this pull request May 24, 2023
…after segment upload to remote store (#7394) (#7718)

Signed-off-by: Ankit Kala <ankikala@amazon.com>
Signed-off-by: Ashish Singh <ssashish@amazon.com>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request May 31, 2023
…cations after segment upload to remote store (opensearch-project#7394) (opensearch-project#7718)"

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <ankikala@amazon.com>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request May 31, 2023
…cations after segment upload to remote store (opensearch-project#7394) (opensearch-project#7718)"

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <ankikala@amazon.com>
ankitkala added a commit to ankitkala/OpenSearch that referenced this pull request May 31, 2023
…cations after segment upload to remote store (opensearch-project#7394) (opensearch-project#7718)"

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <ankikala@amazon.com>
dreamer-89 pushed a commit that referenced this pull request May 31, 2023
…cations after segment upload to remote store (#7394) (#7718)" (#7839)

This reverts commit 92571b7.

Signed-off-by: Ankit Kala <ankikala@amazon.com>
scrawfor99 pushed a commit to scrawfor99/OpenSearch that referenced this pull request May 31, 2023
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Added changelog

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix the test

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* gradle check

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed PR comments

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed comments 3

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix Changelog entry

Signed-off-by: Ankit Kala <ankikala@amazon.com>

---------

Signed-off-by: Ankit Kala <ankikala@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…after segment upload to remote store (opensearch-project#7394)

* SegRep with Remote: Add hook for publishing checkpoint notifications after segment upload to remote store

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Added changelog

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix the test

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* gradle check

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed PR comments

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressing PR comments 2

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Addressed comments 3

Signed-off-by: Ankit Kala <ankikala@amazon.com>

* Fix Changelog entry

Signed-off-by: Ankit Kala <ankikala@amazon.com>

---------

Signed-off-by: Ankit Kala <ankikala@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants