testing: add shuffle flag

[bradfitz]

The testing package should have a -testing.shuffle boolean flag, to randomize the order of tests. This would help shake out accidental ordering assumptions. I'd enable this flag before doing stress testing of Go 1.5 on hundreds of VMs before the release.

[doktorbro]

This is great. I hope it lands in the next version.

[gopherbot]

CL https://golang.org/cl/9635 mentions this issue.

[bradfitz]

We'd also want to make any failures be reproducible later. That implies a PRNG, and printing the seed in test failures. So maybe -testing.shuffle should not be a custom flag type (boolean or seed value), or there should be a second flag to set the seed.

[jasdel]

I'd suggest two flags, a boolean to shuffle with a default seed(time), and a flag which takes a seed as a parameter. When a test fails print out the seed used. I think this would work, because as a user I would only care about the seed if I needed to reproduce a failure case.

[doktorbro]

I like what @jasdel say.

• shuffle (boolean)
• seed (integer)

[josharian]

I'm not sure that we should shuffle benchmarks. Re-arranging benchmarks, particularly microbenchmarks, can result in apparent performance changes: #7920. Also, benchcmp assumes that benchmarks have a stable order. This helps get good results when benchmarks in different packages have the same name. It is also what makes the -best flag possible.

[rsc]

Marking Go1.6Early. I think there's a lot of design left here.

[rsc]

The design did not happen, so not for Go 1.6. If someone is interested, please file a new proposal issue (golang.org/s/proposal). We'll probably need a design doc.