HDDS-6618. Require successful basic checks for long-running tests

[devmadhuu]

What changes were proposed in this pull request?

Since basic checks are fairly quick, so we can postpone starting acceptance, kubernetes and integration checks until these finish. So this PR addressees following 2 things:

• Extract basic(unit) task into a separate job.
• And doesn't start acceptance, kubernetes and integration checks until basic-test(unit) is completed successfully.

Later this basic-test(unit) may be changed to run as part of integration test as basic unit tests also takes significant amount of time to finish.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6618

How was this patch tested?

This patch was tested by running CI jobs with updated ci.yml file.

[adoroszlai]

Thanks @devmadhuu for working on this.

doesn't start acceptance, kubernetes and integration checks until basic-test(unit) is completed successfully

These checks should not wait for unit (which is long-running), rather basic (which includes checkstyle, etc.).

[adoroszlai]

Thanks @devmadhuu for updating the patch.

On second look, I would like to ask for two additional changes:

1. native also runs tests and is somewhat slow, so let's move it from basic to the unit check.
2. With the current implementation (selective_ci_checks.sh outputting the two separate lists of basic/unit checks), moving a check requires too many changes in the script and its test. So I propose keeping selective_ci_checks.* unchanged, and introducing a separate script. This new script would accept the single list of basic checks (including both kinds), and output two lists (one for basic, another one for unit). This way we separate the concerns of "which checks need to be run" and "which basic checks are quick".

[adoroszlai]

Thanks @devmadhuu for iterating on this patch. I think it's almost good to go now. A few minor tweaks suggested.

[adoroszlai]

Can be simplified a bit:

• grep -l only prints paths with match(es), so we can omit the first cut
• basename extracts the filename out of a path
• xargs helps apply a command to each item from stdin

Suggested change

	BASIC_CHECKS=$(grep -r '^#checks:' hadoop-ozone/dev-support/checks \
	| sort -u | cut -f1 -d':' | rev | cut -f1 -d'/' | rev \
	| cut -f1 -d'.')
	BASIC_CHECKS=$(grep -lr '^#checks:' hadoop-ozone/dev-support/checks \
	| sort -u | xargs -n1 basename \
	| cut -f1 -d'.')

The same applies to similar commands in selective_basic_ci_checks.sh.

[devmadhuu]

Thanks @adoroszlai for re-review. I have incorporated your suggestions. Pls check.

[adoroszlai]

This step (and its two outputs) can be added as the last step in build-info, no need for a separate job. The step's input needs to be tweaked a bit to ${{ steps.selective-checks.outputs.basic-checks }}.

[devmadhuu]

Thanks @adoroszlai for re-review. I have incorporated your suggestions. Pls check.

[adoroszlai]

The other script is named selective_ci_checks.sh because it decides which checks to run. This one does not filter the checks, rather categorizes them. So I think a better name for this script and the corresponding CI step would be something like categorize_basic_checks.sh and categorize-basic-checks, respectively.

[devmadhuu]

Thanks @adoroszlai for re-review. I have incorporated your suggestions. Pls check.

[adoroszlai]

Thanks @devmadhuu for updating the patch, LGTM.