Skip to content

testing: benchmarks are context-sensitive #7920

Closed
Closed
testing: benchmarks are context-sensitive#7920
Labels
FrozenDueToAgeUnfortunate

Description

@josharian
josharian
opened on May 1, 2014
What steps will reproduce the problem?

1. Put http://play.golang.org/p/-5mfEhIhC0 (a subset of the math/cmplx benchmarks) in
its own package.
2. Run 'go test -bench=Conj .'
3. Comment out BenchmarkCosh.
4. Run 'go test -bench=Conj .'


What is the expected output? What do you see instead?

Adding/removing an unrelated benchmark should not impact benchmark results.

With all three benchmarks:

BenchmarkConj   2000000000           1.08 ns/op

With only two benchmarks:

BenchmarkConj   2000000000           0.81 ns/op

These measurements are very consistent.


Please use labels and text to provide additional information.

The exact reproduction instructions depend on the version. The instructions above are
for:

go version devel +acf346c00e56 Fri Apr 25 06:44:51 2014 -0700 darwin/amd64

Activity

  1. josharian commented on May 1, 2014

    @josharian
    josharian
    on May 1, 2014
    ContributorAuthor

    Comment 1:

    This can also be triggered by compiler changes that alter the code generated for other
benchmarks, which is how I first encountered this.

  2. ianlancetaylor commented on May 1, 2014

    @ianlancetaylor
    ianlancetaylor
    on May 1, 2014
    Contributor

    Comment 2:

    Can you see if the generated code for the benchmark is the same in both cases?  It's
possible that what you are seeing is the effect of crossing instruction cache lines in a
tight loop.  If that is indeed the case there isn't much the compiler or the testing
framework can do, except serve as a cautionary tale for authors of microbenchmarks.
On the other hand, if the code changes, then there is probably a bug.

    Labels changed: added repo-main, release-go1.3maybe.

  3. josharian commented on May 1, 2014

    @josharian
    josharian
    on May 1, 2014
    ContributorAuthor

    Comment 3:

    The generated code for the benchmark is identical.

  4. ianlancetaylor commented on May 1, 2014

    @ianlancetaylor
    ianlancetaylor
    on May 1, 2014
    Contributor

    Comment 4:

    In that case I don't see what can be done on the Go side.  I'd be happy to hear
suggestions, but basically this is a microbenchmark of trivial code for which the loop
and function call overhead dominate.  It is presumably sensitive to the exact code
alignment.

    Labels changed: added release-none, removed release-go1.3maybe.

    Status changed to Unfortunate.

  5. josharian commented on May 1, 2014

    @josharian
    josharian
    on May 1, 2014
    ContributorAuthor

    Comment 5:

    Yeah. Bummer.
Might loop unrolling provide some stability? That could be done long term by the
compiler, and manually in the short term for key stdlib benchmarks.

  6. josharian commented on May 1, 2014

    @josharian
    josharian
    on May 1, 2014
    ContributorAuthor

    Comment 6:

    Unrolling 8x here causes the delta to shrink. Instead of 1.08 to 0.81 (-25%), it is 0.81
to 0.74 (-10%).
It's hard to be sure that this is attributable directly to the unrolling and not to new
code alignment, but it does appear to help. And unrolling is good for reducing benchmark
overhead anyway.
The question then is whether it's worth the ugliness to do by hand, in the short term.

  7. gopherbot commented on May 2, 2014

    @gopherbot
    gopherbot
    on May 2, 2014
    Contributor

    Comment 7:

    CL https://golang.org/cl/91850043 mentions this issue.
  8. josharian
    added
    Unfortunate
    repo-main
    release-none
    on May 2, 2014
  9. golang
    locked and limited conversation to collaborators on Jun 25, 2016
  10. gopherbot
    added
    FrozenDueToAge
    on Jun 25, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeUnfortunate

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions