Group all.sh components that run on the same platform

[gilles-peskine-arm]

We sometimes spend several minutes instantiating a node, doing a checkout, downloading and spinning up a Docker image, only to run a script that only takes a couple of seconds. The goal of this task is to have the Groovy code group components so that it runs multiple components on the same instantiation.

For example, instead of doing a separate instantiation of Jenkins+git+docker to run each tests/scripts/all.sh check_xxx, we could do a single one for tests/scripts/all.sh "check_*". The goal of this task is to do something like this more generally. There's no specific target, just do a bit more than grouping a few checks.

It's perfectly fine to call all.sh with multiple component names. The reason we're running a single component per instantiation is not to run each component in a clean VM: we used to be happy to run all of all.sh sequentially, apart from the time it took. We're running a single component per instantiation because it was the easy way to parallelize.

It's hard for the Groovy code to know how to group components. We should probably annotate all.sh with some indication of how long each component is expected to take, or how to group them. For example, maybe all.sh --list-components-with-times could print something like

all_u16-check_doxygen_warnings,38
all_u16-test_default_out_of_box,142
all_u16-test_default_cmake_gcc_asan,2040
…

where the number is a nominal cost for the component, and the Groovy code could arrange components in a way to get the total cost per instantiation to be approximately come optimal figure.

Some points to keep in mind:

• We can't just run all the components serially because that takes too long. A few of the longer tests must run on their own.
• Some components require a different platform (currently, a few must run on Ubuntu 16.04, a few must run on 18.04+).

There are comments in all.sh indicating the approximate time some commands take. They are very crude approximations, but they might help. Alternatively we can ignore this look at actual times in a CI run.

Here are figures from one run of a 2.28 nightly test: ci-times.txt. (No script, I did some copy-paste and replace-regexp.) The number in the second column is the number of seconds spend in the docker run pipeline step. This shows that we have a lot of tiny components, it's not just check_*.