Provider cross validation CI runs

[paulidale]

• documentation is added or updated
• tests are added or updated

[paulidale]

Cross validation tests are passing.

[t8m]

I believe this is still testing too much and it would require too much maintenance that has no real effect.

IMO from the released versions we should only test the fips provider against the libcrypto & test suite from all the current branches. The full cross testing should be done only across the tips of the branches.

This is sufficient to test that the fips providers are forwards compatible - this is actually the main use-case we need to support. The test suites from the old branches CANNOT in principle be fully compatible with updated providers, otherwise we would not be ever able to change things in providers. So you cannot ever expect the test suite from the 3.0.0 release (or any other release) to fully pass with new fips providers.

If I count it properly we should have 3 released fips providers * 3 branches + 3*2 branches tests -> 15 tests and you can also drop the old FIPS provider compat test as that is just duplicate.

[paulidale]

I'm basically doing what you suggest. No release is tested against any other release. Nothing is tested against itself.
I set up the jobs to cross test everything because I couldn't figure out a workable alternative. However, the unwanted cases are immediately short-circuited and early exit. E.g. look at this run. I suggest we leave any "additional" tests here, they can be removed once there is a problem.

The old FIPS provider compatibility check is per push not daily, so more timely. Happy to drop it.

[t8m]

I suggest we leave any "additional" tests here, they can be removed once there is a problem.

You already have some hacks there such as skipping the ssl tests from 3.0.0 release. I am afraid they will become unmaintainable at some point. I would simply drop the cases where you're running tests from released version tree.

[paulidale]

Isn't it useful to learn when the 3.1.x FIPS provider ceases working with the 3.1.0 release?
Dropping these cases loses this information.

The test for each version could dropped or limited to branch heads. It should be possible to not rebuild these each run but that's getting into some arcane depths of actions I'd prefer not to touch at the moment.

[t8m]

Isn't it useful to learn when the 3.1.x FIPS provider ceases working with the 3.1.0 release? Dropping these cases loses this information.

I am not sure it is that much useful. I would assume if there is really an incompatibility being introduced the test of 3.0.x branch with the 3.1.x FIPS provider would reveal it as well. And as inevitably this can and will happen you would have to limit the failing test (or patch it somehow) anyway.

[paulidale]

https://github.com/openssl/openssl/actions/runs/4558459775 passing with the reduced testing.

[openssl-machine]

24 hours has passed since 'approval: done' was set, but this PR has failing CI tests. Once the tests pass it will get moved to 'approval: ready to merge' automatically, alternatively please review and set the label manually.

[paulidale]

Trivial edit before merge to remove end of line white space from the new test.
Thanks for the reviews.