HDDS-8640. Support multiple acceptance test scripts with custom config

[adoroszlai]

What changes were proposed in this pull request?

So far each compose environment had a single test script and cluster config. If the same basic environment needs to be tested with custom configuration (e.g. different default bucket type, or using the Vault-based S3 secret store), then the environment is duplicated with minor changes.

The PR updates the acceptance test library (testlib.sh) to allow running more than one test script per environment. Each of these can provide their own configuration and tweaks for the cluster, in a similar way to how the profiling/monitoring extension is implemented for the ozone environment.

This helps avoid duplication, and lets us merge ozone-legacy-bucket into ozone, and ozonesecure-vault into ozonesecure.

https://issues.apache.org/jira/browse/HDDS-8640

How was this patch tested?

CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247

Verified that both test scripts were executed in each of the merged environments:

Executing test ozone/test-legacy-bucket.sh
...
Executing test ozone/test.sh

https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024361030#step:5:47
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024361030#step:5:162

Executing test ozonesecure/test-vault.sh
...
Executing test ozonesecure/test.sh

https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024360968#step:5:47
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024360968#step:5:352

Downloaded test artifacts (unsecure, secure), and verified that cluster logs for each run are separated, but there is a single summary (log.html, etc.):

$ unzip -t acceptance-unsecure.zip                                                                  
Archive:  acceptance-unsecure.zip
    testing: log.html                 OK
    testing: output.log               OK
    testing: ozone-legacy-bucket.xml   OK
    testing: ozone.xml                OK
    testing: ozone/                   OK
    testing: report.html              OK
    testing: summary.html             OK
    testing: ozone/dn-audit-6b41d5ad2bd5.log   OK
    testing: ozone/dn-audit-8587624f6389.log   OK
    testing: ozone/dn-audit-953bb5afb4eb.log   OK
    testing: ozone/dn-audit-ff8406a8f84a.log   OK
    testing: ozone/docker-ozone-ozone-lib-scm.log   OK
    testing: ozone/legacy-bucket/     OK
    testing: ozone/om-audit-97c71ea089d5.log   OK
    testing: ozone/s3g-audit-7b5d545bdf0c.log   OK
    testing: ozone/scm-audit-5e70e8ffb16d.log   OK
    testing: ozone/legacy-bucket/docker-ozone-ozone-recon-nssummary-scm.log   OK
    testing: ozone/legacy-bucket/om-audit-3978fb22b397.log   OK
    testing: ozone/legacy-bucket/scm-audit-0694e8b95722.log   OK

$ unzip -t acceptance-secure.zip  
Archive:  acceptance-secure.zip
    testing: log.html                 OK
    testing: output.log               OK
    testing: ozonesecure-vault.xml    OK
    testing: ozonesecure.xml          OK
    testing: ozonesecure/             OK
    testing: report.html              OK
    testing: summary.html             OK
    testing: ozonesecure/dn-audit-2c8a131019ec.log   OK
    testing: ozonesecure/dn-audit-63a37720394c.log   OK
    testing: ozonesecure/dn-audit-a32fed614881.log   OK
    testing: ozonesecure/docker-ozonesecure-ozonesecure-kinit-scm.log   OK
    testing: ozonesecure/kms-audit.log   OK
    testing: ozonesecure/om-audit-om.log   OK
    testing: ozonesecure/s3g-audit-s3g.log   OK
    testing: ozonesecure/scm-audit-scm.log   OK
    testing: ozonesecure/vault/       OK
    testing: ozonesecure/vault/docker-ozonesecure-ozonesecure-s3-scm.log   OK
    testing: ozonesecure/vault/om-audit-om.log   OK
    testing: ozonesecure/vault/s3g-audit-s3g.log   OK
    testing: ozonesecure/vault/scm-audit-scm.log   OK

Verified that log.html contains the results for both test runs in each environment.

Verified that the clusters for the merged tests had the expected configuration:

$ unzip -j acceptance-unsecure.zip ozone/legacy-bucket/docker-ozone-ozone-recon-nssummary-scm.log
$ grep -o 'ozone.default.bucket.layout=[^,]*' docker-ozone-ozone-recon-nssummary-scm.log | sort -u
ozone.default.bucket.layout=LEGACY

$ unzip -j acceptance-secure.zip ozonesecure/vault/docker-ozonesecure-ozonesecure-s3-scm.log
$ grep -o 'ozone.secret.s3.store.provider=[^,]*' docker-ozonesecure-ozonesecure-s3-scm.log | sort -u 
ozone.secret.s3.store.provider=org.apache.hadoop.ozone.s3.remote.vault.VaultS3SecretStorageProvider

[adoroszlai]

@Pochatkin @xBis7 please take a look

[xBis7]

@adoroszlai That's a very nice improvement. Thanks for working on this.

Can we briefly document the change somewhere like hadoop-ozone/dist/src/main/compose/README.md? There have been many cases where I needed to use some custom configuration and I resorted in a number of tricks.

Something simple like

You can run an acceptance test with specific configuration by creating a .sh file that follows the naming convention test*.sh and exports your custom yaml file with the necessary environment variables. E.g. check legacy-bucket.yaml and test-legacy-bucket.sh.

[adoroszlai]

Can we briefly document the change somewhere like hadoop-ozone/dist/src/main/compose/README.md?

Good point, thanks. I'd like to do it separately (HDDS-8669) to take advantage of quicker CI check for docs.

[xBis7]

@adoroszlai Thanks. I tested it locally by running different configuration setups consecutively and verified adding another shell script won't create any issues and that the environment variables are renewed as expected. Changes LGTM!

[Galsza]

+1 Thank you Attila, these changes are looking good to me.

[sadanand48]

Thanks @adoroszlai for this improvement, the changes look good to me, only thing is we need to follow naming "test-custom-config.sh" and place it in the right directory which is the same comment as above for documenting. +1 .

[adoroszlai]

Thanks @Galsza, @sadanand48, @xBis7 for the review.