Chance of failure in random bias tests is not sufficiently low for CI

[jnm2]

If I knew what I was doing with the binomial confidence interval calculator, 100,000 iterations per test and the ±0.01% tolerance around 50% (same number of ones and zeros) should yield around a 99.9999% success rate, or one failure in a million. Edit: I rounded to 0.01 from 0.0077 and didn't realize that that rounding actually made it better than one failure in a billion, which is 0.0097.

I've seen around ten failures in CI since the PR was merged six hours ago. That means that either I don't know what I'm doing with the math (send help!) or CI has a much worse source of randomness than what I saw at my local machine when I left my computer running them with [Repeat(10000)].

Options

1. Make the bias tests explicit or skip them when an environment variable named CI is present

   • Explicit is not as nice unless we put a note in Randomizer.cs and RandomizerTests.cs to remember to manually run RandomizerBiasTests.cs.

2. See what other people do and maybe get a more standard bias test

3. Get someone with more experience to look at what I've done

@nunit/framework-team Do other options come to mind? I don't really want to delete them, and in fact I want to add 10-100 times more of them at some point.

[jnm2]

Oh, duh, this was supposed to be 100_000 trials, not 10_000 trials:

nunit/src/NUnitFramework/tests/Internal/RandomizerBiasTests.cs

Line 61 in 1a34eac

const int totalCount = 10000;

That increases the chance of failure dramatically and that's also what I'm seeing locally. PR soon!