Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: add shuffle flag #10655

Closed
bradfitz opened this issue May 1, 2015 · 8 comments

Comments

Projects
None yet
6 participants
@bradfitz
Copy link
Member

commented May 1, 2015

The testing package should have a -testing.shuffle boolean flag, to randomize the order of tests. This would help shake out accidental ordering assumptions. I'd enable this flag before doing stress testing of Go 1.5 on hundreds of VMs before the release.

@bradfitz bradfitz added this to the Go1.5Maybe milestone May 1, 2015

@doktorbro

This comment has been minimized.

Copy link

commented May 3, 2015

This is great. I hope it lands in the next version.

@gopherbot

This comment has been minimized.

Copy link

commented May 3, 2015

CL https://golang.org/cl/9635 mentions this issue.

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented May 3, 2015

We'd also want to make any failures be reproducible later. That implies a PRNG, and printing the seed in test failures. So maybe -testing.shuffle should not be a custom flag type (boolean or seed value), or there should be a second flag to set the seed.

@jasdel

This comment has been minimized.

Copy link
Contributor

commented May 4, 2015

I'd suggest two flags, a boolean to shuffle with a default seed(time), and a flag which takes a seed as a parameter. When a test fails print out the seed used. I think this would work, because as a user I would only care about the seed if I needed to reproduce a failure case.

@doktorbro

This comment has been minimized.

Copy link

commented May 4, 2015

I like what @jasdel say.

  • shuffle (boolean)
  • seed (integer)
@josharian

This comment has been minimized.

Copy link
Contributor

commented May 4, 2015

I'm not sure that we should shuffle benchmarks. Re-arranging benchmarks, particularly microbenchmarks, can result in apparent performance changes: #7920. Also, benchcmp assumes that benchmarks have a stable order. This helps get good results when benchmarks in different packages have the same name. It is also what makes the -best flag possible.

@bradfitz bradfitz modified the milestones: Go1.6, Go1.5Maybe May 6, 2015

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jun 26, 2015

Marking Go1.6Early. I think there's a lot of design left here.

@rsc rsc modified the milestones: Go1.6Early, Go1.6 Jun 26, 2015

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 4, 2015

The design did not happen, so not for Go 1.6. If someone is interested, please file a new proposal issue (golang.org/s/proposal). We'll probably need a design doc.

@rsc rsc closed this Nov 4, 2015

@golang golang locked and limited conversation to collaborators Nov 4, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.