Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use more rigourous statistics #4

Open
dgryski opened this issue Jul 24, 2016 · 2 comments
Open

use more rigourous statistics #4

dgryski opened this issue Jul 24, 2016 · 2 comments
Labels

Comments

@dgryski
Copy link

@dgryski dgryski commented Jul 24, 2016

Russ Cox has written https://godoc.org/rsc.io/benchstat which does the same thing but with better math.

@BurntSushi
Copy link
Owner

@BurntSushi BurntSushi commented Jul 24, 2016

Yes, I've seen that and I definitely want to add better math.

@mgeisler
Copy link
Contributor

@mgeisler mgeisler commented May 21, 2017

This might not be necessary. As far as I understand, go test -bench does not do any kind of statistics itself -- it just runs the benchmark code repeated until the running time exceeds 1 sec (by default) and outputs the average: https://golang.org/src/testing/benchmark.go

So this means that the Go world needs extra tools to do statistics on multiple runs. However, that is not the case for cargo bench which comes with some builtin statistics and outlier detection: https://github.com/rust-lang/rust/blob/master/src/libtest/stats.rs#L308

This suggests to me that cargo benchcmp should be careful with comparing values that are already filtered and adjusted by cargo bench -- similar to how an average of an average doesn't give you a proper average of the underlying values. Perhaps cargo bench itself could be improved with more fancy statistic -- I don't know much about that since I'm not a statistician :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.