Allow test clusters to run with TLS #8900

scrawfor99 · 2023-07-26T20:52:17Z

Description

Allow configuration of test clusters to run with TLS.

I copied the change patterns from this PR: https://github.com/opensearch-project/OpenSearch/pull/1051/files. Thank you @VachaShah for the help. From there this is the linked issue about figuring out how to exercise the changes in these files directly: #1086. Seems unaddressed, so hopefully this will not be a blocker.

Related Issues

opensearch-project/security#3056

Check List

~~New functionality includes testing.~~
- ~~All tests pass~~
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

github-actions · 2023-07-26T20:54:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21022/
CommitID: c59a6d8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

github-actions · 2023-07-26T21:06:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21023/
CommitID: e93561d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

peternied

I'm having trouble seeing how the value 'http.protocal' is used, shouldn't there be a change ~line 575 so the WaitForHttpResource object uses the correct protocol?

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

scrawfor99 · 2023-07-27T16:48:10Z

I'm having trouble seeing how the value 'http.protocal' is used, shouldn't there be a change ~line 575 so the WaitForHttpResource object uses the correct protocol?

Yeah, we need to take the value from the settings and use it where you mentioned. The issue is it is not clear how we can configure the tests on the fly and actually change that value. There is a lot of circular logic between the OpenSearchCluster and OpenSearchNode files and most the values appear hard coded. The settings that are passed are all about the file structure which is not what we want. I will have to ask someone more familiar with this test setup because I cannot find a way to actually pass a value on execution.

github-actions · 2023-07-27T16:53:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/21099/
CommitID: b47521c
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

opensearch-trigger-bot · 2023-08-16T21:23:58Z

Compatibility status:

Checks if related components are compatible with change 7ffcd65

Incompatible components

Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-08-16T21:32:27Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22938/
CommitID: c320559
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

dblock

Code looks good. So, how do I use this?

github-actions · 2023-08-16T22:48:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

URL: https://build.ci.opensearch.org/job/gradle-check/22942/
CommitID: c320559
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

scrawfor99 · 2023-08-17T13:16:01Z

Hi @dblock, that is a good question haha. I am adding some notes to using it in the Security repo, but basically it is used with this process: opensearch-project/security#3056 (comment). In short, with this change, you can go into the security repo and grab the code changes that will shortly be merged when I fix them all. The working branch is here: https://github.com/scrawfor99/security/tree/testClusterChanges.

Then you will run the BWC tests bwcTestSuite and provide arguments for the system properties I detailed here. Specifically, it will look something like

./gradlew -p bwc-test clean bwcTestSuite \
   -Dtests.security.manager=false \
   -Dtests.opensearch.http.protocol=https \
   -Dtests.opensearch.username=admin \
   -Dtests.opensearch.password=admin \
   -PcustomDistributionUrl="/Users/steecraw/OpenSearch/distribution/archives/linux-tar/build/distributions/opensearch-min-3.0.0-SNAPSHOT-linux-x64.tar.gz" \
   -i

This lets you specify security information when running our BWC tests so we can then confirm when the tests operate with Security and establish secure connections between the nodes.

dblock · 2023-08-17T13:33:07Z

Can we add this (and any missing) info to https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-backwards-compatibility ~~as part of this PR or a subsequent PR please? You pick?~~ I'll merge, make another PR for TESTING.md?

opensearch-trigger-bot · 2023-08-17T13:35:25Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-8900-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6a5b464c71964800507d389cfb0ae2d358175857
# Push it to GitHub
git push --set-upstream origin backport/backport-8900-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8900-to-2.x.

Yury-Fridlyand

Sorry for the late review. Your changes broke some of my tests in another repo and I started investigation.

Yury-Fridlyand · 2023-08-17T20:10:43Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

+                        getFirstNode().getCredentials().get(0).get("username"),
+                        getFirstNode().getCredentials().get(0).get("password"),
+                        nodes.size()
+                    );
+                    wait.setUsername(getFirstNode().getCredentials().get(0).get("username"));
+                    wait.setPassword(getFirstNode().getCredentials().get(0).get("password"));


Why do you set credentials twice here?

One constructs the URL, one configures the the object.

Yury-Fridlyand · 2023-08-17T20:11:05Z

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java

+                        wait.setUsername(credentials.get(0).get("useradd"));
+                        wait.setPassword(credentials.get(0).get("-p"));


Why useradd and -p and not username and password?

This was support for the previous testing

scrawfor99 · 2023-08-17T23:09:57Z

Sorry for the late review. Your changes broke some of my tests in another repo and I started investigation.

What repo? Do you have an issue you can link?

Yury-Fridlyand · 2023-08-17T23:23:44Z

@scrawfor99
I already found a fix. But you are welcome to review:
Bit-Quill/opensearch-project-sql#335

acarbonetto · 2023-08-18T18:38:02Z

@scrawfor99 updated link for Yury's PR: opensearch-project/sql#1986
Please take a look if you have a moment. This allows us to run IT tests against a cluster & remote cluster that has the security and SQL plugins installed.

Backport of 6a5b464 from #8900 Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>

* Basic idea Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make configurable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update change log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Have to ask around Signed-off-by: Stephen Crawford <steecraw@amazon.com> * add http protocol configuration Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update settings Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure to find Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix assigment Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * prevent empty string Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap optional use Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap back run setup Signed-off-by: Stephen Crawford <steecraw@amazon.com> * try stream fix Signed-off-by: Stephen Crawford <steecraw@amazon.com> * store Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove config mentioons Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd setting to config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * readd overridable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Add setting to env Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Buildable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * update cert passing Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * buildable no logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove uneeded configs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Pass old creds Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix bad logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Reta's requests Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap to boolean Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make bool Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * failed with string logic Signed-off-by: Stephen Crawford <steecraw@amazon.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com>

* Basic idea Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make configurable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update change log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Have to ask around Signed-off-by: Stephen Crawford <steecraw@amazon.com> * add http protocol configuration Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update settings Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure to find Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix assigment Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * prevent empty string Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap optional use Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap back run setup Signed-off-by: Stephen Crawford <steecraw@amazon.com> * try stream fix Signed-off-by: Stephen Crawford <steecraw@amazon.com> * store Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove config mentioons Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd setting to config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * readd overridable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Add setting to env Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Buildable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * update cert passing Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * buildable no logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove uneeded configs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Pass old creds Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix bad logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Reta's requests Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap to boolean Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make bool Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * failed with string logic Signed-off-by: Stephen Crawford <steecraw@amazon.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Gagan Juneja <gjjuneja@amazon.com>

* Basic idea Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make configurable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update change log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Have to ask around Signed-off-by: Stephen Crawford <steecraw@amazon.com> * add http protocol configuration Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update settings Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure to find Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix assigment Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * prevent empty string Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap optional use Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap back run setup Signed-off-by: Stephen Crawford <steecraw@amazon.com> * try stream fix Signed-off-by: Stephen Crawford <steecraw@amazon.com> * store Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove config mentioons Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd setting to config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * readd overridable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Add setting to env Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Buildable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * update cert passing Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * buildable no logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove uneeded configs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Pass old creds Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix bad logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Reta's requests Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap to boolean Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make bool Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * failed with string logic Signed-off-by: Stephen Crawford <steecraw@amazon.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Kiran Reddy <kkreddy@amazon.com>

@scrawfor99

### Description Opening up a PR to describe the issues faced with BWC tests with the security plugin installed and solicit feedback. I plan to forward port this change to main, but first wanted to show this working for 2.9 -> 2.10 tests (as of the time of this writing). Thanks to the work that @scrawfor99 did in [core](opensearch-project/OpenSearch#8900) to supply security settings to testClusters to be able to run the initial wait for cluster yellow checks with a URL that includes the right protocol (`https` when security is enabled) along with a username and password to authenticate the request. I ran into 4 hurdles to get this to run: 1. Initially the cluster didn't form. After a lot of frustration, I ended up finding that by supplying `network.bind_host` and `network.publish_host` to both 127.0.0.1 it resolved the issue. These could probably be combined into a single `network.host`, but I chose to keep them separated. 2. I had issue testing changes to the gradle build-tools after making changes locally. This was the most frustrating hurdle, but ultimately the solution was to change the [`opensearch.version` setting in `bwc-test/build.gradle`](https://github.com/opensearch-project/security/blob/2.x/bwc-test/build.gradle#L47) to `2.10.0-SNAPSHOT`. This value is specifically used as the version of the gradle build-tools that the [BWC tests use](https://github.com/opensearch-project/security/blob/main/bwc-test/build.gradle#L58). The changes I made locally didn't reflect because I was publishing to maven local from the 2.x branch (currently 2.10) and it was looking for 2.9.0-SNAPSHOT artifacts. After updating the value it found my maven local snapshots. For this artifact you can produce maven local snapshots using `./gradlew :build-tools:publishToMavenLocal` from the respective branch in the core repo. 3. After the waitForYellow checks were able to run successfully, the REST Client in the SecurityBackwardsCompatibilityIT was also having problems connecting to the cluster because it didn't recognize the certificates of the server. I ended up using the overly trustworthy route where there is no SSL verification for the REST Client used in this test. I borrowed this implementation from [k-NN's ODFERestTestCase](https://github.com/opensearch-project/k-NN/blob/2.x/src/testFixtures/java/org/opensearch/knn/ODFERestTestCase.java#L118-L141) which is widely used in the plugin ecosystem. There is an open issue to abstract this class into common-utils. More work can be done here to ensure the rest-high-level-client runs with a truststore with the root certificate. 4. The last hurdle I faced was a WarningFailureException where the REST Client could not deserialize the cluster health response because of a warning that was returned with the response about the request including system indices. According to this [comment](opensearch-project/OpenSearch#1108 (comment)), this may only be enabled in snapshots. To fix this, I set preserve cluster to true which [bypasses the method](https://github.com/opensearch-project/OpenSearch/blob/main/test/framework/src/main/java/org/opensearch/test/rest/OpenSearchRestTestCase.java#L364) where the error was thrown. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Enhancement ### Issues Resolved #3056 ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>

@scrawfor99

Opening up a PR to describe the issues faced with BWC tests with the security plugin installed and solicit feedback. I plan to forward port this change to main, but first wanted to show this working for 2.9 -> 2.10 tests (as of the time of this writing). Thanks to the work that @scrawfor99 did in [core](opensearch-project/OpenSearch#8900) to supply security settings to testClusters to be able to run the initial wait for cluster yellow checks with a URL that includes the right protocol (`https` when security is enabled) along with a username and password to authenticate the request. I ran into 4 hurdles to get this to run: 1. Initially the cluster didn't form. After a lot of frustration, I ended up finding that by supplying `network.bind_host` and `network.publish_host` to both 127.0.0.1 it resolved the issue. These could probably be combined into a single `network.host`, but I chose to keep them separated. 2. I had issue testing changes to the gradle build-tools after making changes locally. This was the most frustrating hurdle, but ultimately the solution was to change the [`opensearch.version` setting in `bwc-test/build.gradle`](https://github.com/opensearch-project/security/blob/2.x/bwc-test/build.gradle#L47) to `2.10.0-SNAPSHOT`. This value is specifically used as the version of the gradle build-tools that the [BWC tests use](https://github.com/opensearch-project/security/blob/main/bwc-test/build.gradle#L58). The changes I made locally didn't reflect because I was publishing to maven local from the 2.x branch (currently 2.10) and it was looking for 2.9.0-SNAPSHOT artifacts. After updating the value it found my maven local snapshots. For this artifact you can produce maven local snapshots using `./gradlew :build-tools:publishToMavenLocal` from the respective branch in the core repo. 3. After the waitForYellow checks were able to run successfully, the REST Client in the SecurityBackwardsCompatibilityIT was also having problems connecting to the cluster because it didn't recognize the certificates of the server. I ended up using the overly trustworthy route where there is no SSL verification for the REST Client used in this test. I borrowed this implementation from [k-NN's ODFERestTestCase](https://github.com/opensearch-project/k-NN/blob/2.x/src/testFixtures/java/org/opensearch/knn/ODFERestTestCase.java#L118-L141) which is widely used in the plugin ecosystem. There is an open issue to abstract this class into common-utils. More work can be done here to ensure the rest-high-level-client runs with a truststore with the root certificate. 4. The last hurdle I faced was a WarningFailureException where the REST Client could not deserialize the cluster health response because of a warning that was returned with the response about the request including system indices. According to this [comment](opensearch-project/OpenSearch#1108 (comment)), this may only be enabled in snapshots. To fix this, I set preserve cluster to true which [bypasses the method](https://github.com/opensearch-project/OpenSearch/blob/main/test/framework/src/main/java/org/opensearch/test/rest/OpenSearchRestTestCase.java#L364) where the error was thrown. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Enhancement opensearch-project#3056 - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>

@scrawfor99

### Description Opening up a PR to describe the issues faced with BWC tests with the security plugin installed and solicit feedback. Thanks to the work that @scrawfor99 did in [core](opensearch-project/OpenSearch#8900) to supply security settings to testClusters to be able to run the initial wait for cluster yellow checks with a URL that includes the right protocol (`https` when security is enabled) along with a username and password to authenticate the request. I ran into 4 hurdles to get this to run: 1. Initially the cluster didn't form. After a lot of frustration, I ended up finding that by supplying `network.bind_host` and `network.publish_host` to both 127.0.0.1 it resolved the issue. These could probably be combined into a single `network.host`, but I chose to keep them separated. 2. I had issue testing changes to the gradle build-tools after making changes locally. This was the most frustrating hurdle, but ultimately the solution was to change the [`opensearch.version` setting in `bwc-test/build.gradle`](https://github.com/opensearch-project/security/blob/2.x/bwc-test/build.gradle#L47) to `2.10.0-SNAPSHOT`. This value is specifically used as the version of the gradle build-tools that the [BWC tests use](https://github.com/opensearch-project/security/blob/main/bwc-test/build.gradle#L58). The changes I made locally didn't reflect because I was publishing to maven local from the 2.x branch (currently 2.10) and it was looking for 2.9.0-SNAPSHOT artifacts. After updating the value it found my maven local snapshots. For this artifact you can produce maven local snapshots using `./gradlew :build-tools:publishToMavenLocal` from the respective branch in the core repo. 3. After the waitForYellow checks were able to run successfully, the REST Client in the SecurityBackwardsCompatibilityIT was also having problems connecting to the cluster because it didn't recognize the certificates of the server. I ended up using the overly trustworthy route where there is no SSL verification for the REST Client used in this test. I borrowed this implementation from [k-NN's ODFERestTestCase](https://github.com/opensearch-project/k-NN/blob/2.x/src/testFixtures/java/org/opensearch/knn/ODFERestTestCase.java#L118-L141) which is widely used in the plugin ecosystem. There is an open issue to abstract this class into common-utils. More work can be done here to ensure the rest-high-level-client runs with a truststore with the root certificate. 4. The last hurdle I faced was a WarningFailureException where the REST Client could not deserialize the cluster health response because of a warning that was returned with the response about the request including system indices. According to this [comment](opensearch-project/OpenSearch#1108 (comment)), this may only be enabled in snapshots. To fix this, I set preserve cluster to true which [bypasses the method](https://github.com/opensearch-project/OpenSearch/blob/main/test/framework/src/main/java/org/opensearch/test/rest/OpenSearchRestTestCase.java#L364) where the error was thrown. * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Enhancement ### Issues Resolved #3056 ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: Craig Perkins <cwperx@amazon.com>

* Basic idea Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make configurable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update change log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Have to ask around Signed-off-by: Stephen Crawford <steecraw@amazon.com> * add http protocol configuration Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update settings Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure to find Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix assigment Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * prevent empty string Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap optional use Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap back run setup Signed-off-by: Stephen Crawford <steecraw@amazon.com> * try stream fix Signed-off-by: Stephen Crawford <steecraw@amazon.com> * store Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove config mentioons Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd setting to config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * readd overridable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Add setting to env Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Buildable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * update cert passing Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * buildable no logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove uneeded configs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Pass old creds Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix bad logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Reta's requests Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap to boolean Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make bool Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * failed with string logic Signed-off-by: Stephen Crawford <steecraw@amazon.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

* Basic idea Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make configurable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update change log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Have to ask around Signed-off-by: Stephen Crawford <steecraw@amazon.com> * add http protocol configuration Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update settings Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure to find Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix assigment Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * prevent empty string Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap optional use Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap back run setup Signed-off-by: Stephen Crawford <steecraw@amazon.com> * try stream fix Signed-off-by: Stephen Crawford <steecraw@amazon.com> * store Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove config mentioons Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd setting to config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * readd overridable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Add setting to env Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Buildable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * update cert passing Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * buildable no logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove uneeded configs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Pass old creds Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix bad logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Reta's requests Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap to boolean Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make bool Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * failed with string logic Signed-off-by: Stephen Crawford <steecraw@amazon.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

* Basic idea Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make configurable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update change log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Have to ask around Signed-off-by: Stephen Crawford <steecraw@amazon.com> * add http protocol configuration Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update settings Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * retry integ Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Fix failure to find Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix assigment Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * set default Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * prevent empty string Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap optional use Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap back run setup Signed-off-by: Stephen Crawford <steecraw@amazon.com> * try stream fix Signed-off-by: Stephen Crawford <steecraw@amazon.com> * store Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove config mentioons Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix failure Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Readd setting to config Signed-off-by: Stephen Crawford <steecraw@amazon.com> * readd overridable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Add setting to env Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Buildable Signed-off-by: Stephen Crawford <steecraw@amazon.com> * update cert passing Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove log Signed-off-by: Stephen Crawford <steecraw@amazon.com> * buildable no logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove uneeded configs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Pass old creds Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix bad logs Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Reta's requests Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Swap to boolean Signed-off-by: Stephen Crawford <steecraw@amazon.com> * trigger retry Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Make bool Signed-off-by: Stephen Crawford <steecraw@amazon.com> * fix npe Signed-off-by: Stephen Crawford <steecraw@amazon.com> * spotless Signed-off-by: Stephen Crawford <steecraw@amazon.com> * failed with string logic Signed-off-by: Stephen Crawford <steecraw@amazon.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Basic idea

c59a6d8

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

scrawfor99 mentioned this pull request Jul 26, 2023

Backward compatibility tests do not run with the security plugin enabled opensearch-project/security#3056

Closed

6 tasks

scrawfor99 added 2 commits July 26, 2023 16:56

Make configurable

e93561d

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

Update change log

485ea16

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

peternied reviewed Jul 26, 2023

View reviewed changes

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java Outdated Show resolved Hide resolved

buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java Show resolved Hide resolved

Have to ask around

b47521c

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

add http protocol configuration

df95f72

Signed-off-by: Stephen Crawford <steecraw@amazon.com>

scrawfor99 marked this pull request as ready for review July 27, 2023 18:25

scrawfor99 requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja and dreamer-89 as code owners July 27, 2023 18:25

dblock approved these changes Aug 16, 2023

View reviewed changes

dblock merged commit 6a5b464 into opensearch-project:main Aug 17, 2023
10 of 11 checks passed

dblock added the backport 2.x Backport to 2.x branch label Aug 17, 2023

opensearch-trigger-bot bot added the backport-failed label Aug 17, 2023

scrawfor99 mentioned this pull request Aug 17, 2023

Add Secure Bwc documentation to TESTING.md #9414

Merged

6 tasks

Yury-Fridlyand reviewed Aug 17, 2023

View reviewed changes

scrawfor99 mentioned this pull request Aug 18, 2023

[Backport 2.x] Allow test clusters to run with TLS #8900 #9444

Merged

6 tasks

cwperks mentioned this pull request Aug 28, 2023

[2.x] Enable security for bwc tests opensearch-project/security#3257

Merged

3 tasks

cwperks mentioned this pull request Aug 30, 2023

Enable security for bwc tests opensearch-project/security#3269

Merged

3 tasks

jmazanec15 mentioned this pull request Dec 19, 2023

Add option to set user for test cluster to perform initial health checks #11641

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow test clusters to run with TLS #8900

Allow test clusters to run with TLS #8900

scrawfor99 commented Jul 26, 2023 •

edited

github-actions bot commented Jul 26, 2023

github-actions bot commented Jul 26, 2023

peternied left a comment

scrawfor99 commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

opensearch-trigger-bot bot commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

dblock left a comment

github-actions bot commented Aug 16, 2023

scrawfor99 commented Aug 17, 2023

dblock commented Aug 17, 2023 •

edited

opensearch-trigger-bot bot commented Aug 17, 2023

Yury-Fridlyand left a comment

Yury-Fridlyand Aug 17, 2023

scrawfor99 Aug 17, 2023

Yury-Fridlyand Aug 17, 2023

scrawfor99 Aug 17, 2023

scrawfor99 commented Aug 17, 2023

Yury-Fridlyand commented Aug 17, 2023

acarbonetto commented Aug 18, 2023

		wait.setUsername(credentials.get(0).get("useradd"));
		wait.setPassword(credentials.get(0).get("-p"));

Allow test clusters to run with TLS #8900

Allow test clusters to run with TLS #8900

Conversation

scrawfor99 commented Jul 26, 2023 • edited

Description

Related Issues

Check List

github-actions bot commented Jul 26, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jul 26, 2023

Gradle Check (Jenkins) Run Completed with:

peternied left a comment

Choose a reason for hiding this comment

scrawfor99 commented Jul 27, 2023

github-actions bot commented Jul 27, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 16, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

dblock left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with:

scrawfor99 commented Aug 17, 2023

dblock commented Aug 17, 2023 • edited

opensearch-trigger-bot bot commented Aug 17, 2023

Yury-Fridlyand left a comment

Choose a reason for hiding this comment

Yury-Fridlyand Aug 17, 2023

Choose a reason for hiding this comment

scrawfor99 Aug 17, 2023

Choose a reason for hiding this comment

Yury-Fridlyand Aug 17, 2023

Choose a reason for hiding this comment

scrawfor99 Aug 17, 2023

Choose a reason for hiding this comment

scrawfor99 commented Aug 17, 2023

Yury-Fridlyand commented Aug 17, 2023

acarbonetto commented Aug 18, 2023

scrawfor99 commented Jul 26, 2023 •

edited

dblock commented Aug 17, 2023 •

edited