Adding support for integration tests with remote cluster.

[saratvemulapalli]

Signed-off-by: Sarat Vemulapalli vemulapallisarat@gmail.com

Description

Adding support for running integration tests with remote cluster.

Plugin integration tests depend on opensearch.testclusters gradle plugin exported by OpenSearch. Plugins use the same framework to run integration tests against a remote cluster. The problem is it internally downloads the right distribution to setup the test cluster even though it's not used.

Until now that worked since the min artifact is published for every release.
In order to step up our testing flow as part of the release, we now would like to run integration tests before the artifacts are published to verify the bundle via https://github.com/opensearch-project/opensearch-build#integration-tests. But since the artifact is not published the tests would see:

Before:

* What went wrong:
Execution failed for task ':integTest'.
> Could not resolve all files for configuration ':opensearch_distro_extracted_testclusters--integTest-0-1.1.0-'.
   > Could not resolve opensearch-distribution:opensearch:1.1.0.
     Required by:
         project :
      > Could not resolve opensearch-distribution:opensearch:1.1.0.
         > Could not get resource 'https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz'. Received status code 403 from server: Forbidden
      > Could not resolve opensearch-distribution:opensearch:1.1.0.
         > Could not get resource 'https://artifacts.opensearch.org/release-candidates/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz'.
            > Could not HEAD 'https://artifacts.opensearch.org/release-candidates/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz'. Received status code 403 from server: Forbidden

After:

% ./gradlew integTestRemote -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername="docker-cluster" -Dhttps=true -Duser=admin -Dpassword=admin
82 tests completed, 12 failed, 1 skipped

So this PR adds support to run integration tests for remote clusters without the min artifacts being published at artifacts.opensearch.org.

Issues Resolved

opensearch-project/OpenSearch#1240
opensearch-project/opensearch-plugins#101

Check List

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #298 (2f4f964) into main (29c6e08) will decrease coverage by 0.38%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #298      +/-   ##
============================================
- Coverage     76.31%   75.93%   -0.39%     
+ Complexity     3963     3950      -13     
============================================
  Files           295      295              
  Lines         17165    17165              
  Branches       1812     1812              
============================================
- Hits          13100    13034      -66     
- Misses         3236     3299      +63     
- Partials        829      832       +3     

Flag	Coverage Δ
plugin	75.93% <ø> (-0.39%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...java/org/opensearch/ad/task/ADBatchTaskRunner.java	75.22% <0.00%> (-6.54%)	⬇️
...port/SearchAnomalyDetectorInfoTransportAction.java	60.00% <0.00%> (-4.45%)	⬇️
...rch/ad/transport/ForwardADTaskTransportAction.java	94.06% <0.00%> (-3.39%)	⬇️
...ain/java/org/opensearch/ad/task/ADTaskManager.java	75.43% <0.00%> (-1.61%)	⬇️
...ava/org/opensearch/ad/task/ADHCBatchTaskCache.java	88.88% <0.00%> (-1.24%)	⬇️
.../opensearch/ad/transport/StopDetectorResponse.java	42.85% <0.00%> (+23.80%)	⬆️

[ohltyler]

@saratvemulapalli is it reasonable for this to be skipped for 1.2 specifically? Only concern is backporting to 1.2 branch can delay the bundle builds more.

[saratvemulapalli]

@ohltyler how much of an effort/time is it to run the builds again?
This will help avoid running manual tests for the release for the plugin owners.
cc: @peternied @peterzhuamazon

[ohltyler]

@saratvemulapalli considering we already have bug fixes backported to 1.2 today, we can add this as well if it gets merged today. Changes make sense to me.

[peterzhuamazon]

@ohltyler how much of an effort/time is it to run the builds again? This will help avoid running manual tests for the release for the plugin owners. cc: @peternied @peterzhuamazon

Build is automatic every 2 hours for around 20+10 = 30min for tar + docker.
So I think I agree with @ohltyler to approve and merge.

Thanks.