testing: decide whether benchmark probe should use b.N==0 or b.N==1

[rsc]

We need some kind of probe phase to gather sub-benchmark information.
It would be nice if that phase were clearly identified, for example by b.N==0.
Some benchmarks have tests that assume there was at least one iteration,
and those would need updating. We've now updated the standard library.

This is something people would have to update tests for, but of course non-test
programs continue working fine. And the testing package has never guaranteed
that b.N > 0. We do break code that depends on things it shouldn't, for example
when we updated sort.Sort in Go 1.6. But we don't want to break too much.

If, based on beta testing, we decide that there are just too many benchmarks
in real-world code that will need updating, then we should back off to using N==1
as the subtest probe.

Decide based on beta testing though, not by guessing now.

[bradfitz]

We need some kind of probe phase to gather sub-benchmark information.

Is that statement already jumping to conclusions? Why can't the Benchmark signature be different and take more than just a *testing.B for benchmarks which want subtests?

(To save other clueless readers time, this is about bug #12166 with design doc https://github.com/golang/proposal/blob/master/design/12166-subtests.md)

[minux]

I want to ask the same question. When a new feature breaks old code, there seems like enough reason to redesign the new feature to not break existing code (esp. when the standard library code is also broken.) Why subbenchmarks have to be registered with a regular benchmark function?

[mpvl]

On Saturday, March 19, 2016, Minux Ma notifications@github.com wrote:

I want to ask the same question. When a new feature breaks old code, there
seems like enough reason to redesign the new feature to not break existing
code (esp. when the standard library code is also broken.) Why
subbenchmarks have to be registered with a regular benchmark function?

Please see
https://github.com/golang/proposal/blob/master/design/12166-subtests.md

The feature can easily be made to not break these tests. The breaking
version is a bit nicer so this is just to see if we can make things nicer
without much effort. If not, no problem.
Note that the breakage is very rare (but easy to identify) and occurs only
in quite unconventional uses of the benchmark functionality that are in
addition making use of an undocumented and not guaranteed property of B.

—
You are receiving this because you were assigned.
Reply to this email directly or view it on GitHub
#14863 (comment)

[gopherbot]

CL https://golang.org/cl/21043 mentions this issue.

[mpvl]

Trade-offs:
Advantage of N=0: more principled, allows easy way for tests to do a setup once.

Disadvantage: less compatible: some code expects N>1 and code may be called a different number of times from before.

Overall, we don't allow benchmarks to do both measurements and run subtests. So we can piggyback on the old N==1 phase to discover if there are subbenchmarks without changing the processing flow.

So overall, using the current N==1 as the "probe" phase is probably better. Moreover, to do setup once, a user can use a flag, or even use subtests:

func BenchmarkFoo(b *B) {
    // setup
   b.Run(b *B) {
      // doBench
   }
}