Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: benchmarks are context-sensitive #7920

Closed
josharian opened this issue May 1, 2014 · 7 comments

Comments

Projects
None yet
3 participants
@josharian
Copy link
Contributor

commented May 1, 2014

What steps will reproduce the problem?

1. Put http://play.golang.org/p/-5mfEhIhC0 (a subset of the math/cmplx benchmarks) in
its own package.
2. Run 'go test -bench=Conj .'
3. Comment out BenchmarkCosh.
4. Run 'go test -bench=Conj .'


What is the expected output? What do you see instead?

Adding/removing an unrelated benchmark should not impact benchmark results.

With all three benchmarks:

BenchmarkConj   2000000000           1.08 ns/op

With only two benchmarks:

BenchmarkConj   2000000000           0.81 ns/op

These measurements are very consistent.


Please use labels and text to provide additional information.

The exact reproduction instructions depend on the version. The instructions above are
for:

go version devel +acf346c00e56 Fri Apr 25 06:44:51 2014 -0700 darwin/amd64
@josharian

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2014

Comment 1:

This can also be triggered by compiler changes that alter the code generated for other
benchmarks, which is how I first encountered this.
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 1, 2014

Comment 2:

Can you see if the generated code for the benchmark is the same in both cases?  It's
possible that what you are seeing is the effect of crossing instruction cache lines in a
tight loop.  If that is indeed the case there isn't much the compiler or the testing
framework can do, except serve as a cautionary tale for authors of microbenchmarks.
On the other hand, if the code changes, then there is probably a bug.

Labels changed: added repo-main, release-go1.3maybe.

@josharian

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2014

Comment 3:

The generated code for the benchmark is identical.
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

commented May 1, 2014

Comment 4:

In that case I don't see what can be done on the Go side.  I'd be happy to hear
suggestions, but basically this is a microbenchmark of trivial code for which the loop
and function call overhead dominate.  It is presumably sensitive to the exact code
alignment.

Labels changed: added release-none, removed release-go1.3maybe.

Status changed to Unfortunate.

@josharian

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2014

Comment 5:

Yeah. Bummer.
Might loop unrolling provide some stability? That could be done long term by the
compiler, and manually in the short term for key stdlib benchmarks.
@josharian

This comment has been minimized.

Copy link
Contributor Author

commented May 1, 2014

Comment 6:

Unrolling 8x here causes the delta to shrink. Instead of 1.08 to 0.81 (-25%), it is 0.81
to 0.74 (-10%).
It's hard to be sure that this is attributable directly to the unrolling and not to new
code alignment, but it does appear to help. And unrolling is good for reducing benchmark
overhead anyway.
The question then is whether it's worth the ugliness to do by hand, in the short term.
@gopherbot

This comment has been minimized.

Copy link

commented May 2, 2014

Comment 7:

CL https://golang.org/cl/91850043 mentions this issue.

@golang golang locked and limited conversation to collaborators Jun 25, 2016

This issue was closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.