The testing package should have a -testing.shuffle boolean flag, to randomize the order of tests. This would help shake out accidental ordering assumptions. I'd enable this flag before doing stress testing of Go 1.5 on hundreds of VMs before the release.
The text was updated successfully, but these errors were encountered:
We'd also want to make any failures be reproducible later. That implies a PRNG, and printing the seed in test failures. So maybe -testing.shuffle should not be a custom flag type (boolean or seed value), or there should be a second flag to set the seed.
I'd suggest two flags, a boolean to shuffle with a default seed(time), and a flag which takes a seed as a parameter. When a test fails print out the seed used. I think this would work, because as a user I would only care about the seed if I needed to reproduce a failure case.
I'm not sure that we should shuffle benchmarks. Re-arranging benchmarks, particularly microbenchmarks, can result in apparent performance changes: #7920. Also, benchcmp assumes that benchmarks have a stable order. This helps get good results when benchmarks in different packages have the same name. It is also what makes the -best flag possible.