testing: run first benchmark with last value from -cpu list GOMAXPROCS value

[KochankovID]

What version of Go are you using (go version)?

$ go version
go version go1.17.4 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/ilyakochankov/Library/Caches/go-build"
GOENV="/Users/ilyakochankov/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/ilyakochankov/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/ilyakochankov/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.17.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/ilyakochankov/Yandex.Disk.localized/University/4_cource/Программирование на ГПУ/homework1/task_3/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/tl/2sfc4zys07sfhnn129zy0l4c0000gn/T/go-build2189710306=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I wrote a simple function for matrix multiplication with gorutines:

type Matrix32 struct {
	Height, Width int
	Data          []float32
}

func NewMatrix(height, width int) Matrix32 {
	return Matrix32{height, width, make([]float32, height*width)}
}

func (m *Matrix32) FillRand() {
	rand.Seed(time.Now().Unix())
	for i := range m.Data {
		m.Data[i] = rand.Float32()
	}
}

func ParallelMatMul(a, b *Matrix32) Matrix32 {
	numThreads := runtime.GOMAXPROCS(0)
	fmt.Println(numThreads) // for debug purpose

	if a.Width < numThreads {
		numThreads = a.Width
	}

	result := NewMatrix(a.Height, b.Width)
	batchSize := result.Width / numThreads

	var wg sync.WaitGroup
	for theadNum := 0; theadNum < numThreads; theadNum++ {
		wg.Add(1)
		go func(wg *sync.WaitGroup, theadNum int) {
			defer wg.Done()

			for x := theadNum * batchSize; x < (theadNum+1)*batchSize; x++ {
				for y := 0; y < result.Height; y++ {
					for i := 0; i < a.Width; i++ {
						result.Data[x*result.Height+y] += a.Data[y+i*a.Height] * b.Data[x*b.Height+i]
					}
				}
			}
		}(&wg, theadNum)
	}

Also, I wrote a benchmark test for this function:

func BenchmarkParallelMatMulOnHeap(b *testing.B) {
	xHeight, xWidth := 512*2, 512*4
	yHeight, yWidth := 512*4, 512*2

	x, y := task_3.NewMatrix(xHeight, xWidth), task_3.NewMatrix(yHeight, yWidth)
	x.FillRand()
	y.FillRand()

	b.ResetTimer()
	task_3.ParallelMatMul(&x, &y)
}

And I want to run it with command:

go test ./... -bench=BenchmarkParallelMatMulOnHeap -benchmem -cpu 1,2,4  -run=^#

What did you expect to see?

It looks like that first benchmark runs with 4 threads except 1 (as I expected). And so the results of first and last benchmarks are the same.

I suppose it happens because my benchmark takes long time, so it executes only one time (if I make size of the matrices smaller it would execute several times and results will be correct). It's probably happens because here:

        testRan, testOk := runTests(m.deps.MatchString, m.tests, deadline)
	exampleRan, exampleOk := runExamples(m.deps.MatchString, m.examples)
	m.stopAlarm()
	if !testRan && !exampleRan && *matchBenchmarks == "" {
		fmt.Fprintln(os.Stderr, "testing: warning: no tests to run")
	}
	if !testOk || !exampleOk || !runBenchmarks(m.deps.ImportPath(), m.deps.MatchString, m.benchmarks) || race.Errors() > 0 {
		fmt.Println("FAIL")
		m.exitCode = 1
		return
	}

In runTests GOMAXPROCS set in cycle. But in the first run of benchmarks, GOMAXPROCS is'n changed.

What did you see instead?

?       taks_3/cmd/task_3       [no test files]
?       taks_3/internal/app/task_3      [no test files]

Num theads: 4
goos: darwin
goarch: arm64
pkg: taks_3/test
BenchmarkParallelMatMulOnHeap                  1        2309626959 ns/op         4196728 B/op         10 allocs/op
BenchmarkParallelMatMulOnHeap-2         
Num theads: 2
       1        4447989458 ns/op         4194904 B/op          7 allocs/op
BenchmarkParallelMatMulOnHeap-4         
Num theads: 4
       1        2370282750 ns/op         4195480 B/op          7 allocs/op
PASS
ok      taks_3/test     9.508s