Need infrastructure to run tests in varying hardware intrinsic modes

[danmoseley]

Increasingly we are taking changes that use hardware intrinsics to accelerate parts of CoreFX. Without special care, our testing will only ever cover the AVX2 (or AVX) path, not the software path and not any SSE path.

We need a way to easily cause selected tests to re-run in each mode so every codepath is exercised.

In the ML.NET repo they repeat relevant tests under each of regular, no AVX, and no AVX/SSE using RemoteExec:
https://github.com/dotnet/machinelearning/blob/d65af0f755b9cd5739693004f1a8a8964ce23d8a/test/Microsoft.ML.CpuMath.UnitTests.netcoreapp/UnitTests.cs#L140

We can do something similar, but centralize the magic environment variables and offer a flag on RemoteExec that tests can set to enumerate over them.

cc @tannergooding @jkotas @Anipik

[ViktorHofer]

Moving to future as this would be nice to have but not necessarily important for 3.0. That said I will prioritize it correctly and try to work on it as soon as other high prio items are done. I'll try to do at least a manual run asap to get some data.

[gfoidl]

It would be nice to write tests as

[Fact]
[HwIntrinsic(Intrinsic.Avx2 | Intrinsic.Ssse3)]
public void MySuperTest()
{
    // ...
}

and then the test-runtime / CI executes the tests with environment variables set (i.e. that the noted intrinsics are disabled) so that the code-path can be verified.

The approach with RemoteExec is possible, but it forces us to write some code that could belong to the infrastructure.

[danmoseley]

Both approaches above assume it's fairly well defined which tests we need to run this way. As more and more of our code uses intrinsics and particularly if we want some indirect test coverage we might instead want to execute the entire test suite periodically with the various environment variables.

Perhaps it could be added pairwise into our test matrix, eg., on Centos and Windows Server always go through software mode, etc - if we're confident it's an orthogonal parameter.

[danmoseley]

linking to discussion of what the knobs are/should be: #11701

[danmoseley]

@stephentoub @jkotas now that we're being more thoughtful about our matrix, and more of it is post-commit, would it be reasonable to pairwise one or more of these flags into the mix? Eg., if we test Windows 10 x64 and Windows 11 x64 we could enable COMPlus_EnableHWIntrinsic=0 permanently on one of those. It seems to me vanishingly unlikely that that would mask any OS-specific bugs.

It's not entirely clear which switch/es to set though. COMPlus_EnableHWIntrinsic=0 will force the software path, which in some cases is (still?) exercised for free on Arm64 tests. It may be better to force the SSE2 path, eg.

cc @safern @tannergooding @BruceForstall