testing: benchmarks timing issue when using -count > 1

[ALTree]

@laboger reported in #25520 that 05ca340 introduced timing issues in the results of some benchmarks.

For example, the ChanNonblocking benchmark in runtime is much slower after the 1st iteration on ppc64le

BenchmarkChanNonblocking-16 1000000000 3.15 ns/op
BenchmarkChanNonblocking-16 1000000000 32.7 ns/op

The benchmark uses RunParallel. It seems the the change linked above may have introduced timing issues on benchmarks that use RunParallel. For example, using this:

package p_test

import (
	"bytes"
	"testing"
	"text/template"
)

func BenchmarkF(b *testing.B) {
	templ := template.Must(template.New("test").Parse("Hello, {{.}}!"))
	b.RunParallel(func(pb *testing.PB) {
		var buf bytes.Buffer
		for pb.Next() {
			buf.Reset()
			templ.Execute(&buf, "World")
		}
	})
}

I see on go1.10:

BenchmarkF-4   	10000000	       188 ns/op
BenchmarkF-4   	10000000	       190 ns/op
BenchmarkF-4   	10000000	       189 ns/op
BenchmarkF-4   	10000000	       188 ns/op
BenchmarkF-4   	10000000	       189 ns/op

tip:

BenchmarkF-4   	10000000	       187 ns/op
BenchmarkF-4   	10000000	       198 ns/op
BenchmarkF-4   	10000000	       195 ns/op
BenchmarkF-4   	10000000	       195 ns/op
BenchmarkF-4   	10000000	       194 ns/op

notice how on tip every run after the 1st consistently reports an higher ns/op.

[erifan]

The same issue on arm64:
goos: linux
goarch: arm64
pkg: encoding/json
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 3.51 ns/op
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 77.9 ns/op
BenchmarkTypeFieldsCache/HitTypes1-32 500000000 64.9 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 3.55 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 64.6 ns/op
BenchmarkTypeFieldsCache/HitTypes10-32 500000000 63.6 ns/op

[ALTree]

I looked into this yesterday and I got nowhere. I'll send a revert for that change.

[gopherbot]

Change https://golang.org/cl/115535 mentions this issue: Revert "testing: only compute b.N once when passed -count > 1"

[ALTree]

The change was reverted. @laboger @erifan if you could verify that the bench regression is gone on ppc64le and arm64 that would be great.

[laboger]

I noticed this went in and tried it right away -- looks like it is now working as expected for ppc64le.

[erifan]

It is now working as expected for arm64.