[FEATURE] Add support for SNAPSHOT distributions

[dbwiddis]

Is your feature request related to a problem?

The distributionUrl requires the URL of a tar distribution. The Extensions SDK is under active development including the need to test performance fixes. When setting up clusters to test Extensions (see opensearch-project/opensearch-sdk-java#652) there is a need to test OpenSearch from the latest Snapshot build. These zip files are published for integration tests here

However, the CDK hardcodes the .tgz filename to retrieve, and the unpacking method, here:

opensearch-cluster-cdk/lib/infra/infra-stack.ts

Lines 398 to 402 in b450ca4

	InitCommand.shellCommand(`set -ex;mkdir opensearch; curl -L ${props.distributionUrl} -o opensearch.tar.gz;`
	+ 'tar zxf opensearch.tar.gz -C opensearch --strip-components=1; chown -R ec2-user:ec2-user opensearch;', {
	cwd: '/home/ec2-user',
	ignoreErrors: false,
	}),

What solution would you like?

Allow a user to specify these snapshot filenames via a URL parameter. Specifically:

• allow user to change opensearch.tar.gz to opensearch-3.0.0-20230509.201738-827.zip
• use unzip rather than tar zxf to extract from the archive.

What alternatives have you considered?

Using the SDK to create a cluster, and insert a script in infra-stack.ts to download from git version control and build the distro locally.

Do you have any additional context?

This is probably more complicated than the brief description above.

[dbwiddis]

Updating this request:

• An acceptable workaround for SNAPSHOT is to use the nightly build
• This is apparently already automatically done for plugins when the distVersion is provided
• Suggestion is that distributionUrl not be required; if missing, will use the "latest" resovled from distVersion.
  • alternately, some text for the url such as "latest" that serves this same purpose; could also apply to the dashboardsUrl

[rishabh6788]

At the moment we only support tar.gz extension and nightly snapshots are also available in the same format see https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz.
We also have nightly benchmark runs for 2.x and 3.x snapshots.