[FEATURE] Run integration tests as part of the OpenSearch distribution

[dblock]

Is your feature request related to a problem?
Following a problem in another plugin (opensearch-project/opensearch-build#2043), and coming from opensearch-project/opensearch-build#58, there's no automated testing that this plugin runs as part of the OpenSearch distribution.

What solution would you like?
Run integration tests as part of the distribution.

[ankitkala]

Just to confirm, do we want to add the jenkins workflow per plugin to run the integration tests? Or is the ask to implement a generic script (like test.sh) as an entry point which will be invoked to run the tests from distribution build workflow on existing clusters.

[manishdev-amzn]

This requires brining up 2 clusters and cross cluster integration script will use these enndpoints for testing. To address this, build system team will require to make changes in bringing up the clusters and cross cluster team will makes changes to integration tests. We are targeting CCR repo level changes provided the build level changes are in place . Can build system team provide the build infra level changes? CC: @dblock @CEHENKLE

[manishdev-amzn]

Related issue: #244 (comment)

[Pallavi-AWS]

@dblock we will add basic testcase to check sanity of distribution build by checking plugin installation. This can be done by invoking a plugin API (_start, follower_stats) and validate failure. Any functionality around replication still needs more than one cluster in the framework, which we will manually verify when docker artifacts are ready (@ankitkala, @saikaranam-amazon)

[dblock]

@Pallavi-AWS I assigned this to you so we don't drop it, please reassign this issue and #244 as you see fit.

[dblock]

@Pallavi-AWS The multiple cluster work is opensearch-project/opensearch-build#1499, child of opensearch-project/opensearch-build#1428

[ankitkala]

@dblock The multi cluster support mentioned above is for performance tests where we're spinning up the clusters on dedicated ec2 nodes using CDK.
I think for integ tests, we rely on LocalTestCluster: https://github.com/opensearch-project/opensearch-build/blob/main/src/test_workflow/integ_test/integ_test_suite_opensearch.py#L67