Run xfel-regression on Azure

[Anthchirp]

Adds the xfel-regression test pipeline laid out in #1671 to pull request builds only, and in such a way as to

1. minimize total build time, and
2. separating out regular test failures from xfel-regression test failures

At this time this is work in progress, especially to test out the pull-request build behaviour, but on merging will also close #1671.

[Anthchirp]

New issue after this is merged:

This dials test is not run during as part of the main tests, as xfel_regression is not available at this point:

SKIPPED [1] test/algorithms/integration/test_kapton.py:20: test requires xfel_regression

It is also not run as part of the PR builds where xfel_regression is available, as it is not part of the xfel_regression test suite.
So we never run this test here.

I would suggest that either the test is moved into xfel_regression, or the test data is moved into dials-data. At ~6.6 MB the latter shouldn't really be an issue, all it really needs is a description file.

[Anthchirp]

This is now ready for review.

xfel-regression tests are run after the DIALS tests passed, and in PR builds only. The output is aggregated and reported in a separate build job. I also added another PR job stage ("Summary: Ready to merge") that summarizes the cross-platform dials tests without the xfel-regression tests.

I note that the xfel-regression tests currently fail on MacOS, I'm not sure whether they are supposed to, or not. We can disable the xfel-regression test on MacOS for now and address this in a separate issue/PR.

[phyy-nx]

Thanks! This is great, appreciate the effort.

The failure on macos is the cosym test which uses MPI (It does pass on linux). The cctbx_project macos Azure XFEL CI pipeline skips the test as it doesn't install mpi4py. I've essentially never done MPI stuff on macos, so I'd say it's safe to skip that test on macos.

The rest of the tests pass on macos so I'll just add a macos check to the run_tests.py in xfel_regression.

I'll work on test_kapton.py separately.

[phyy-nx]

Ok, cosym test updated to only run on linux. Ready to re-run the tests, but I'm not sure if I have the access to do so.

[phyy-nx]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Anthchirp]

Glad to hear!

(The easiest way to force a rerun of the PR build only is to just close and reopen the pull request.)

[phyy-nx]

Gotcha thanks. Looks like a clean build!

[Anthchirp]

kapton stuff -> #1712, will merge this here