New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use more rigourous statistics #4

Open
dgryski opened this Issue Jul 24, 2016 · 2 comments

Comments

Projects
None yet
3 participants
@dgryski

dgryski commented Jul 24, 2016

Russ Cox has written https://godoc.org/rsc.io/benchstat which does the same thing but with better math.

@BurntSushi

This comment has been minimized.

Show comment
Hide comment
@BurntSushi

BurntSushi Jul 24, 2016

Owner

Yes, I've seen that and I definitely want to add better math.

Owner

BurntSushi commented Jul 24, 2016

Yes, I've seen that and I definitely want to add better math.

@mgeisler

This comment has been minimized.

Show comment
Hide comment
@mgeisler

mgeisler May 21, 2017

Contributor

This might not be necessary. As far as I understand, go test -bench does not do any kind of statistics itself -- it just runs the benchmark code repeated until the running time exceeds 1 sec (by default) and outputs the average: https://golang.org/src/testing/benchmark.go

So this means that the Go world needs extra tools to do statistics on multiple runs. However, that is not the case for cargo bench which comes with some builtin statistics and outlier detection: https://github.com/rust-lang/rust/blob/master/src/libtest/stats.rs#L308

This suggests to me that cargo benchcmp should be careful with comparing values that are already filtered and adjusted by cargo bench -- similar to how an average of an average doesn't give you a proper average of the underlying values. Perhaps cargo bench itself could be improved with more fancy statistic -- I don't know much about that since I'm not a statistician :-)

Contributor

mgeisler commented May 21, 2017

This might not be necessary. As far as I understand, go test -bench does not do any kind of statistics itself -- it just runs the benchmark code repeated until the running time exceeds 1 sec (by default) and outputs the average: https://golang.org/src/testing/benchmark.go

So this means that the Go world needs extra tools to do statistics on multiple runs. However, that is not the case for cargo bench which comes with some builtin statistics and outlier detection: https://github.com/rust-lang/rust/blob/master/src/libtest/stats.rs#L308

This suggests to me that cargo benchcmp should be careful with comparing values that are already filtered and adjusted by cargo bench -- similar to how an average of an average doesn't give you a proper average of the underlying values. Perhaps cargo bench itself could be improved with more fancy statistic -- I don't know much about that since I'm not a statistician :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment