HDDS-4000. Split acceptance tests to reduce CI feedback time

[adoroszlai]

What changes were proposed in this pull request?

Reduce total CI time:

1. Split acceptance tests into 3 groups executed in parallel using GitHub Actions matrix build. Groups are defined to have similar run times. misc has several unrelated items, some of which can be moved to another group if it grows too long.

• secure: 50m 36s
• unsecure: 48m 23s
• misc: 50m 6s

2. Merge some integration test groups, both to make them similar length and to reduce the number of checks. Currently some groups have much shorter run time. Since client is the longest one, others can take more time, too. New run times:

• client: 54m 37s
• filesystem-hdds: 43m 58s
• ozone: 55m 37s

3. Start integration checks earlier: do not wait for build check to finish. Both acceptance and integration checks perform a full build anyway. The only reason to wait was that if there is a compile error, we wanted to avoid spawning too many checks that fail for the same reason. But compile error is the least frequent CI problem, checkstyle, rat and various test failures are much more common. So being optimistic helps in the most common scenarios. (Ideally we should cache the results of build for later stages. Then we could both depend on it and avoid duplicate builds.)

https://issues.apache.org/jira/browse/HDDS-4000

How was this patch tested?

https://github.com/adoroszlai/hadoop-ozone/runs/897297500

[elek]

Nice idea (and nice Jira number BTW ;-) ).

I like this tagging system especially because it can support optional tests. For example, I can create some optional ..._test.sh files with specific tags (like restart-tests) which are excluded by default pr runs but run by the daily builds.

It can be easy to forget adding #suite:misc to a new file. What do you think to support execution all the test files without suite tags. (--> misc is the default suite)?

[adoroszlai]

I like this tagging system especially because it can support optional tests. For example, I can create some optional ..._test.sh files with specific tags (like restart-tests) which are excluded by default pr runs but run by the daily builds.

Note that they still need to be named exactly test.sh.

It can be easy to forget adding #suite:misc to a new file.

Who writes such files from scratch? :)

What do you think to support execution all the test files without suite tags. (--> misc is the default suite)?

Agree, this is more safe.

[codecov-commenter]

Codecov Report

Merging #1236 into master will decrease coverage by 0.05%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1236      +/-   ##
============================================
- Coverage     73.73%   73.68%   -0.06%     
+ Complexity    10124    10113      -11     
============================================
  Files           974      974              
  Lines         50123    50123              
  Branches       4881     4881              
============================================
- Hits          36958    36932      -26     
- Misses        10835    10858      +23     
- Partials       2330     2333       +3     

Impacted Files	Coverage Δ	Complexity Δ
...e/hadoop/ozone/recon/tasks/OMUpdateEventBatch.java	75.00% <0.00%> (-8.34%)	6.00% <0.00%> (-1.00%)
.../transport/server/ratis/ContainerStateMachine.java	71.07% <0.00%> (-5.39%)	62.00% <0.00%> (-4.00%)
...ozone/container/ozoneimpl/ContainerController.java	63.15% <0.00%> (-5.27%)	11.00% <0.00%> (-1.00%)
.../apache/hadoop/ozone/protocolPB/OzonePBHelper.java	90.00% <0.00%> (-5.00%)	6.00% <0.00%> (-1.00%)
.../common/volume/RoundRobinVolumeChoosingPolicy.java	80.95% <0.00%> (-4.77%)	5.00% <0.00%> (-1.00%)
...e/commandhandler/CreatePipelineCommandHandler.java	87.23% <0.00%> (-4.26%)	8.00% <0.00%> (ø%)
...iner/common/transport/server/ratis/CSMMetrics.java	70.76% <0.00%> (-3.08%)	20.00% <0.00%> (-1.00%)
.../statemachine/background/BlockDeletingService.java	75.73% <0.00%> (-1.48%)	12.00% <0.00%> (-1.00%)
...apache/hadoop/ozone/client/io/KeyOutputStream.java	79.58% <0.00%> (-1.25%)	46.00% <0.00%> (-2.00%)
...oop/ozone/recon/tasks/ReconTaskControllerImpl.java	92.38% <0.00%> (-0.96%)	29.00% <0.00%> (-1.00%)
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4f7e32...d4e13e1. Read the comment docs.

[elek]

+1

Thanks the updated @adoroszlai

BTW, I don't know why the overall time (from 1:10) is increased recently. Do you have any information about that?

[adoroszlai]

Thanks @elek for reviewing and committing it.

BTW, I don't know why the overall time (from 1:10) is increased recently. Do you have any information about that?

We now have 4x FS tests ((o3fs, ofs) x (bucket, link)) and 2x S3 tests (bucket, link). Upgrade is also new.