Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
testing: add shuffle flag #10655
I'd suggest two flags, a boolean to shuffle with a default seed(time), and a flag which takes a seed as a parameter. When a test fails print out the seed used. I think this would work, because as a user I would only care about the seed if I needed to reproduce a failure case.
I'm not sure that we should shuffle benchmarks. Re-arranging benchmarks, particularly microbenchmarks, can result in apparent performance changes: #7920. Also, benchcmp assumes that benchmarks have a stable order. This helps get good results when benchmarks in different packages have the same name. It is also what makes the