Fixing test flakiness

[sleepy-owl]

The test test_ground_truth_separated_modes sometimes fails non-deterministically. This PR addresses this issue.

To find a solution, I collected samples of value of statistic in the assertion from several test executions and computed the tail distribution. I computed the extreme percentiles to check how high can the values be. The computed percentiles are as follows:

0.9:: 0.09
0.99 :: 0.16
0.999:: 0.20
0.9999 :: 0.30

For this fix, i chose the 99.9th percentile. I think setting the bound using the statistical evaluation might be a good way to ensure the test is not flaky.

Do you guys think this makes sense? Please let me know if this looks good or if you have any other suggestions. Also, here I assume there are no bugs in the code under test.

[yannikschaelte]

Hi @sleepy-owl , thanks for this contribution! I think checking the test percentiles is the way to go indeed (unless we set the RNG, which we however rather don't want to atm). The Kolmogorov-Smirnov test is unfortunately rather unstable, s.t. the 20% may be the best we can do here I guess.

[sleepy-owl]

Thanks! Can you please merge this if this looks good?

[sleepy-owl]

@yannikschaelte gentle ping! Is there anything else that I should include in the PR? if not, can you please merge the PR?

[yannikschaelte]

Was waiting for another PR, but will merge soon, thanks!

[sleepy-owl]

Thanks!