Disallow multiple data paths for search nodes #6427

xuezhou25 · 2023-02-22T00:27:15Z

Description

At node bootstrap, check if the node has search role and multiple data paths, if yes, fail the bootstrap.

Issues Resolved

resolve #6274

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

andrross · 2023-02-22T00:47:31Z

server/src/test/java/org/opensearch/bootstrap/BootstrapChecksTests.java

@@ -774,4 +779,22 @@ Version getVersion() {
        version.set(Runtime.version());
        BootstrapChecks.check(emptyContext, true, Collections.singletonList(check));
    }
+
+    public void testMultipleDataPathCheck() throws NodeValidationException {


I think there should be 3 cases here:

search node, 1 data path

search node, 2 data paths

data node, 2 data paths

You have case 2 covered, but not really the other two. They should be separate test methods as well because it is generally a best practice to make each test case test one thing whenever possible.

Got it! Thanks for the reviews Andrew :)

andrross · 2023-02-22T00:48:42Z

CHANGELOG.md

@@ -101,6 +101,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix timeout error when adding a document to an index with extension running ([#6275](https://github.com/opensearch-project/OpenSearch/pull/6275))
 - Handle translog upload during primary relocation for remote-backed indexes ([#5804](https://github.com/opensearch-project/OpenSearch/pull/5804))
 - Batch translog sync/upload per x ms for remote-backed indexes ([#5854](https://github.com/opensearch-project/OpenSearch/pull/5854))
+- Add bootstrap check to avoid search node has multiple data paths ([#6427](https://github.com/opensearch-project/OpenSearch/pull/6427))


I'd call this "Disallow multiple data paths for search nodes". Please update the commit message and PR title as well.

andrross · 2023-02-22T00:49:05Z

server/src/main/java/org/opensearch/bootstrap/BootstrapChecks.java

+    /**
+     * Bootstrap check that if a search node contains multiple data paths
+     */
+    static class MultipleDataPathCheck implements BootstrapCheck {


Can this be private?

Why should this class be private? I think it is reasonable to be default

The answer to my question is "no" because it is referenced in the unit test, which I missed when I made this comment :)

github-actions · 2023-02-22T00:57:42Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11551/
CommitID: e4c972e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-22T01:16:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11553/
CommitID: d1a7b87
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dreamer-89 · 2023-02-22T05:34:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/11553/

CommitID: d1a7b87
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Looks like a legit failure @xuezhou25

REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.bootstrap.BootstrapChecksTests.testMultipleDataPathCheck" -Dtests.seed=ED64E1E8740B72F -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=sr -Dtests.timezone=Europe/Budapest -Druntime.java=19

org.opensearch.bootstrap.BootstrapChecksTests > testMultipleDataPathCheck FAILED
    java.lang.AssertionError: 
    Expected: a string containing "Having multiple data paths in the search role is not allowed"
         but: was "[1] bootstrap checks failed
    [1]: Having multiple data paths in the search node is not allowed"
        at __randomizedtesting.SeedInfo.seed([ED64E1E8740B72F:A2E0146960C4CA10]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.junit.Assert.assertThat(Assert.java:930)
        at org.opensearch.bootstrap.BootstrapChecksTests.testMultipleDataPathCheck(BootstrapChecksTests.java:796)

dreamer-89 · 2023-02-22T05:36:18Z

server/src/test/java/org/opensearch/bootstrap/BootstrapChecksTests.java

+        );
+        final List<BootstrapCheck> checks = Collections.singletonList(new BootstrapChecks.MultipleDataPathCheck());
+        final NodeValidationException e = expectThrows(NodeValidationException.class, () -> BootstrapChecks.check(context, true, checks));
+        assertThat(e.getMessage(), containsString("Having multiple data paths in the search role is not allowed"));


nit: assertThat -> assertEquals

I think it is preferable to use assertThat due to #2503 (review)

Isn't assertThat deprecated?

org.junit.Assert.assertThat() is deprecated (which OpenSearchTestCase ultimately inherits from). org.hamcrest.MatcherAssert.assertThat is not deprecated and is the intended replacement for that.

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

…a_path # Conflicts: # CHANGELOG.md

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

github-actions · 2023-02-23T07:11:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/11623/
CommitID: 8633846
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-02-23T07:35:05Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11624/
CommitID: 411e7a5

codecov-commenter · 2023-02-23T07:37:38Z

Codecov Report

Merging #6427 (c9ff4d1) into main (4dd0c5f) will decrease coverage by 0.06%.
The diff coverage is 87.50%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #6427      +/-   ##
============================================
- Coverage     70.76%   70.70%   -0.06%     
+ Complexity    59093    59056      -37     
============================================
  Files          4802     4800       -2     
  Lines        282793   282719      -74     
  Branches      40782    40765      -17     
============================================
- Hits         200111   199906     -205     
- Misses        66271    66420     +149     
+ Partials      16411    16393      -18

Impacted Files	Coverage Δ
...va/org/opensearch/cluster/routing/RoutingNode.java	`78.53% <ø> (-0.26%)`	⬇️
...ting/allocation/allocator/LocalShardsBalancer.java	`85.13% <ø> (-0.10%)`	⬇️
...r/routing/allocation/allocator/ShardsBalancer.java	`33.33% <ø> (+8.33%)`	⬆️
...search/common/settings/AbstractScopedSettings.java	`88.38% <ø> (+1.25%)`	⬆️
...rg/opensearch/common/settings/ClusterSettings.java	`91.89% <ø> (ø)`
...n/java/org/opensearch/common/settings/Setting.java	`86.13% <ø> (-0.22%)`	⬇️
...java/org/opensearch/bootstrap/BootstrapChecks.java	`89.87% <85.71%> (+0.71%)`	⬆️
.../allocation/allocator/BalancedShardsAllocator.java	`91.17% <85.71%> (+0.60%)`	⬆️
...egations/support/MultiTermsValuesSourceConfig.java	`80.32% <100.00%> (ø)`
...egations/support/MultiValuesSourceFieldConfig.java	`92.50% <100.00%> (ø)`
... and 510 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

gashutos · 2023-02-23T16:49:20Z

server/src/test/java/org/opensearch/bootstrap/BootstrapChecksTests.java

+        final List<BootstrapCheck> checks = Collections.singletonList(new BootstrapChecks.MultipleDataPathCheck());
+        final NodeValidationException e = expectThrows(NodeValidationException.class, () -> BootstrapChecks.check(context, true, checks));
+        assertThat(e.getMessage(), containsString("Having multiple data paths in the search node is not allowed"));
+    }


Would this kind of private method help to remove some duplicate effort ?

public void performDataPathsCheck(String[] paths, String roleName) throws NodeValidationException { final BootstrapContext context = createTestContext( Settings.builder() .putList(NodeRoleSettings.NODE_ROLES_SETTING.getKey(), Collections.singletonList(roleName)) .putList(Environment.PATH_DATA_SETTING.getKey(), paths) .build(), Metadata.EMPTY_METADATA ); final List<BootstrapCheck> checks = Collections.singletonList(new BootstrapChecks.MultipleDataPathCheck()); BootstrapChecks.check(emptyContext, true, checks); }

andrross

Looking pretty good! Just a couple minor issues.

andrross · 2023-02-23T18:56:46Z

server/src/main/java/org/opensearch/bootstrap/BootstrapChecks.java

+        public BootstrapCheckResult check(BootstrapContext context) {
+            if (NodeRoleSettings.NODE_ROLES_SETTING.get(context.settings()).contains(DiscoveryNodeRole.SEARCH_ROLE)
+                && Environment.PATH_DATA_SETTING.get(context.settings()).size() > 1) {
+                return BootstrapCheckResult.failure("Having multiple data paths in the search node is not allowed");


Another nitpick on the wording, but something like "Multiple data paths are not allowed for search nodes" is probably a bit clearer and more direct.

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

github-actions · 2023-02-23T21:13:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11662/
CommitID: ec4a3d6

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

github-actions · 2023-02-23T22:14:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/11665/
CommitID: c9ff4d1

Add bootstrap check to avoid search node has multiple data path Signed-off-by: Xue Zhou <xuezhou@amazon.com> (cherry picked from commit ceb3928) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Add bootstrap check to avoid search node has multiple data path (cherry picked from commit ceb3928) Signed-off-by: Xue Zhou <xuezhou@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

xuezhou25 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, adnapibar, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and VachaShah as code owners February 22, 2023 00:27

xuezhou25 added 2 commits February 21, 2023 16:42

Add bootstrap check to avoid search node has multiple data path

1207b53

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

Add changelog

d1a7b87

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

xuezhou25 force-pushed the Prevent_search_node_startup_for_multiple_data_path branch from e4c972e to d1a7b87 Compare February 22, 2023 00:45

andrross reviewed Feb 22, 2023

View reviewed changes

dreamer-89 reviewed Feb 22, 2023

View reviewed changes

xuezhou25 changed the title ~~Add bootstrap check to avoid search node has multiple data paths~~ Disallow multiple data paths for search nodes Feb 23, 2023

xuezhou25 added 2 commits February 22, 2023 22:16

Modify comments

6a3cb90

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

Merge branch 'main' into Prevent_search_node_startup_for_multiple_dat…

8633846

…a_path # Conflicts: # CHANGELOG.md

opensearch-project deleted a comment from github-actions bot Feb 23, 2023

fix spotless check

411e7a5

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

gashutos reviewed Feb 23, 2023

View reviewed changes

andrross reviewed Feb 23, 2023

View reviewed changes

Modify comments

ec4a3d6

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

Remove duplication

c9ff4d1

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

kotwanikunal approved these changes Feb 23, 2023

View reviewed changes

andrross approved these changes Feb 23, 2023

View reviewed changes

andrross merged commit ceb3928 into opensearch-project:main Feb 23, 2023

andrross added the backport 2.x Backport to 2.x branch label Feb 23, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 23, 2023

[Backport 2.x] Disallow multiple data paths for search nodes #6472

Merged

xuezhou25 deleted the Prevent_search_node_startup_for_multiple_data_path branch February 24, 2023 01:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disallow multiple data paths for search nodes #6427

Disallow multiple data paths for search nodes #6427

xuezhou25 commented Feb 22, 2023

andrross Feb 22, 2023

xuezhou25 Feb 22, 2023 •

edited

andrross Feb 22, 2023

andrross Feb 22, 2023

xuezhou25 Feb 23, 2023

andrross Feb 23, 2023

github-actions bot commented Feb 22, 2023

github-actions bot commented Feb 22, 2023

dreamer-89 commented Feb 22, 2023 •

edited

Gradle Check (Jenkins) Run Completed with:

dreamer-89 Feb 22, 2023

xuezhou25 Feb 23, 2023

kotwanikunal Feb 23, 2023

andrross Feb 23, 2023

github-actions bot commented Feb 23, 2023

github-actions bot commented Feb 23, 2023

codecov-commenter commented Feb 23, 2023 •

edited

gashutos Feb 23, 2023 •

edited

andrross left a comment

andrross Feb 23, 2023

github-actions bot commented Feb 23, 2023

github-actions bot commented Feb 23, 2023

Disallow multiple data paths for search nodes #6427

Disallow multiple data paths for search nodes #6427

Conversation

xuezhou25 commented Feb 22, 2023

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

xuezhou25 Feb 22, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 22, 2023

Gradle Check (Jenkins) Run Completed with:

dreamer-89 commented Feb 22, 2023 • edited

Gradle Check (Jenkins) Run Completed with:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 23, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Feb 23, 2023 • edited

Codecov Report

gashutos Feb 23, 2023 • edited

Choose a reason for hiding this comment

andrross left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Feb 23, 2023

Gradle Check (Jenkins) Run Completed with:

xuezhou25 Feb 22, 2023 •

edited

dreamer-89 commented Feb 22, 2023 •

edited

codecov-commenter commented Feb 23, 2023 •

edited

gashutos Feb 23, 2023 •

edited