[Segment Replication] Handle failover in mixed cluster mode #9536

Poojita-Raj · 2023-08-24T16:34:43Z

Description

This change will ensure that the replica that's on the oldest opensearch version is promoted to primary in the event of a failover in a segment replication enabled cluster. This would avoid the problem of replicas on older opensearch versions not being able to read the segments sent by newer opensearch version primaries.

Related Issues

Resolves #9027

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-24T17:04:27Z

Compatibility status:

Checks if related components are compatible with change c93d086

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-24T17:21:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23417/
CommitID: c93d086

codecov · 2023-08-24T17:22:24Z

Codecov Report

Merging #9536 (6f31434) into main (012c4fa) will increase coverage by 0.81%.
Report is 10 commits behind head on main.
The diff coverage is 84.00%.

@@             Coverage Diff              @@
##               main    #9536      +/-   ##
============================================
+ Coverage     70.40%   71.21%   +0.81%     
- Complexity    56861    57582     +721     
============================================
  Files          4781     4781              
  Lines        271231   271247      +16     
  Branches      39599    39600       +1     
============================================
+ Hits         190947   193166    +2219     
+ Misses        63974    61934    -2040     
+ Partials      16310    16147     -163

Files Changed	Coverage Δ
...opensearch/action/get/TransportMultiGetAction.java	`44.64% <50.00%> (+1.00%)`	⬆️
.../org/opensearch/action/get/TransportGetAction.java	`65.30% <83.33%> (-0.70%)`	⬇️
...a/org/opensearch/cluster/routing/RoutingNodes.java	`84.04% <83.33%> (-1.28%)`	⬇️
...java/org/opensearch/cluster/metadata/Metadata.java	`88.77% <100.00%> (+0.19%)`	⬆️

... and 585 files with indirect coverage changes

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java

server/src/main/java/org/opensearch/action/get/TransportGetAction.java

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

github-actions · 2023-08-29T18:11:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsOnRemoteStoreRestore

URL: https://build.ci.opensearch.org/job/gradle-check/23739/
CommitID: dc41002
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/action/get/TransportMultiGetAction.java

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

github-actions · 2023-08-29T21:40:47Z

Compatibility status:

Checks if related components are compatible with change 16c4751

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-29T21:52:00Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23757/
CommitID: 16c4751
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Poojita-Raj · 2023-08-29T22:31:59Z

Gradle Check (Jenkins) Run Completed with:

* **RESULT:** FAILURE ❌

* **URL:** https://build.ci.opensearch.org/job/gradle-check/23757/

* **CommitID:** [16c4751](https://github.com/opensearch-project/OpenSearch/commit/16c4751ea64abb36755d281cd11a7655215d9f5f)
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure [a flaky test](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) unrelated to your change?

Flaky test: #9580

github-actions · 2023-08-30T05:05:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23789/
CommitID: 16c4751

server/src/main/java/org/opensearch/cluster/metadata/Metadata.java

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

github-actions · 2023-08-30T17:56:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23845/
CommitID: 6f31434
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T18:01:47Z

Compatibility status:

Checks if related components are compatible with change 6f31434

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

Poojita-Raj · 2023-08-30T18:41:48Z

Gradle Check (Jenkins) Run Completed with:

* **RESULT:** FAILURE ❌

* **URL:** https://build.ci.opensearch.org/job/gradle-check/23845/

* **CommitID:** [6f31434](https://github.com/opensearch-project/OpenSearch/commit/6f31434f858ed0ca2e0c2526e7ced374a3452ade)
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure [a flaky test](https://github.com/opensearch-project/OpenSearch/blob/main/DEVELOPER_GUIDE.md#flaky-tests) unrelated to your change?

#9598

github-actions · 2023-08-30T20:44:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23866/
CommitID: 6f31434
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-30T22:43:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/23878/
CommitID: 6f31434

* pick oldest OS version replica to promote as primary Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add test Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor to avoid coupling Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add comments Signed-off-by: Poojita Raj <poojiraj@amazon.com> --------- Signed-off-by: Poojita Raj <poojiraj@amazon.com> (cherry picked from commit ff65403) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…9652) * pick oldest OS version replica to promote as primary * add test * refactor * refactor to avoid coupling * add comments --------- (cherry picked from commit ff65403) Signed-off-by: Poojita Raj <poojiraj@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ch-project#9536) * pick oldest OS version replica to promote as primary Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add test Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor to avoid coupling Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add comments Signed-off-by: Poojita Raj <poojiraj@amazon.com> --------- Signed-off-by: Poojita Raj <poojiraj@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

…ch-project#9536) * pick oldest OS version replica to promote as primary Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add test Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor to avoid coupling Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add comments Signed-off-by: Poojita Raj <poojiraj@amazon.com> --------- Signed-off-by: Poojita Raj <poojiraj@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

…ch-project#9536) * pick oldest OS version replica to promote as primary Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add test Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor Signed-off-by: Poojita Raj <poojiraj@amazon.com> * refactor to avoid coupling Signed-off-by: Poojita Raj <poojiraj@amazon.com> * add comments Signed-off-by: Poojita Raj <poojiraj@amazon.com> --------- Signed-off-by: Poojita Raj <poojiraj@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Poojita-Raj added the skip-changelog label Aug 24, 2023

kotwanikunal reviewed Aug 25, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java Outdated Show resolved Hide resolved

dreamer-89 requested changes Aug 27, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/get/TransportGetAction.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java Outdated Show resolved Hide resolved

Poojita-Raj added 2 commits August 29, 2023 09:39

add test

b71276e

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

refactor

dc41002

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

Poojita-Raj force-pushed the handleFailover branch from c93d086 to dc41002 Compare August 29, 2023 17:23

Poojita-Raj mentioned this pull request Aug 29, 2023

[Segment Replication] Support realtime TermVector requests with Segment Replication #9585

Merged

6 tasks

mch2 reviewed Aug 29, 2023

View reviewed changes

server/src/main/java/org/opensearch/action/get/TransportMultiGetAction.java Outdated Show resolved Hide resolved

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java Outdated Show resolved Hide resolved

refactor to avoid coupling

16c4751

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

dreamer-89 reviewed Aug 30, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/Metadata.java Show resolved Hide resolved

dreamer-89 reviewed Aug 30, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java Outdated Show resolved Hide resolved

dreamer-89 reviewed Aug 30, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/routing/RoutingNodes.java Show resolved Hide resolved

dreamer-89 approved these changes Aug 30, 2023

View reviewed changes

add comments

6f31434

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

mch2 approved these changes Aug 30, 2023

View reviewed changes

mch2 merged commit ff65403 into opensearch-project:main Aug 31, 2023
12 checks passed

mch2 added the backport 2.x Backport to 2.x branch label Aug 31, 2023

opensearch-trigger-bot bot mentioned this pull request Aug 31, 2023

[Backport 2.x] [Segment Replication] Handle failover in mixed cluster mode #9652

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Segment Replication] Handle failover in mixed cluster mode #9536

[Segment Replication] Handle failover in mixed cluster mode #9536

Poojita-Raj commented Aug 24, 2023

github-actions bot commented Aug 24, 2023

github-actions bot commented Aug 24, 2023

codecov bot commented Aug 24, 2023 •

edited

github-actions bot commented Aug 29, 2023

github-actions bot commented Aug 29, 2023

github-actions bot commented Aug 29, 2023

Poojita-Raj commented Aug 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

Poojita-Raj commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

github-actions bot commented Aug 30, 2023

[Segment Replication] Handle failover in mixed cluster mode #9536

[Segment Replication] Handle failover in mixed cluster mode #9536

Conversation

Poojita-Raj commented Aug 24, 2023

Description

Related Issues

Check List

github-actions bot commented Aug 24, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 24, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 24, 2023 • edited

Codecov Report

github-actions bot commented Aug 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 29, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 29, 2023

Gradle Check (Jenkins) Run Completed with:

Poojita-Raj commented Aug 29, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Poojita-Raj commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 30, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 24, 2023 •

edited