CI Adds quicker CI failure to reduce resource usage

[thomasjpfan]

Reference Issues/PRs

Closes #21403

What does this implement/fix? Explain your changes.

This PR allows Ubuntu_Bionic to run first and have most of the other instances depend on it.

1. I added a second Ubuntu_Bionic_Parallel that runs only when [azure all-parallel] is in the commit message. If [azure all-parallel] is not in the commit message, then Ubuntu_Bionic_Parallel is skipped.

2. Ubuntu_Bionic only runs when [azure all-parallel] is not in the commit message.

3. The other instances are updated to run when Ubuntu_Bionic succeeds or skipped, which covers all the non-failure cases.

[ogrisel]

Could you please add a top level comment in this file to explain the logic behind this dependency graph?

[ogrisel]

Instead of introducing this redundant build entry, we could just change the condition on the other builds to use a dis-junction such as:

or (
    in(dependencies['Ubuntu_Bionic']['result'], 'Succeeded', 'Skipped'),
    contains(dependencies['git_commit']['outputs']['commit.message'], '[azure all-parallel]')
)

[thomasjpfan]

I am unable to get this to work without the redundant build. The other builds are configured with dependsOn: [git_commit, linting, Ubuntu_Bionic], which waits for Ubuntu_Bionic to complete.

[ogrisel]

I think we should use Linux_Latest pylatest_conda_forge_mkl as the default run which is the fastest (runs in 15min total). We might even make it run even faster by using openblas instead of MKL which is slow to download (not sure about the impact of OpenBLAS vs MKL on the tests and use MKL in another linux run.

Edit: Scratch this last comment. On the latest run the bionic job was the fastest.

[thomasjpfan]

After looking at this PR again, I reworked the comments in the yaml files to better explain what is going on. Here are the output of the CI:

• Azure CI with the azure parallel flag: Ubuntu_Jammy_Jellyfish_Parallel job runs in parallel with the other jobs:

• Azure CI without the flag, the Ubuntu_Jammy_Jellyfish runs first and the other jobs are waiting for Ubuntu_Jammy_Jellyfish to succeed:

[jjerphan]

LGTM. The comments are much simpler and clearer to me know. Thank you for making the CI usage less resources, @thomasjpfan.

[ogrisel]

LGTM, thanks for the improvement. Hopefully this will reduce our CI usage.

If this strategy proves to be annoying or too complex to understand, we can always revert.