testing: inconsistent behaviors between running tests directly and after compiling the code first

[rainydew]

What version of Go are you using (go version)?

$ go version
go version go1.20.3 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/baozhenchen/Library/Caches/go-build"
GOENV="/Users/baozhenchen/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/baozhenchen/.gvm/pkgsets/go1.20.3/global/pkg/mod"
GONOPROXY="git.garena.com"
GONOSUMDB="git.garena.com"
GOOS="darwin"
GOPATH="/Users/baozhenchen/.gvm/pkgsets/go1.20.3/global"
GOPRIVATE="git.garena.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/baozhenchen/.gvm/gos/go1.20.3"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/Users/baozhenchen/.gvm/gos/go1.20.3/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.20.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/baozhenchen/gits/go-tester/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/6r/b8pq0wys5gg6j9l59fx0_gfc0000gq/T/go-build2381900583=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

with this code

package gotest_parallel_bug

import (
	"sync"
	"testing"
	"time"
)

func TestParallelBug(t *testing.T) {
	wg := &sync.WaitGroup{}

	t.Run("a", func(t *testing.T) {
		wg.Add(1)
		defer wg.Done()
		t.Parallel()
		time.Sleep(10 * time.Second)
	})

	t.Run("b", func(t *testing.T) {
		wg.Add(1)
		defer wg.Done()
		t.Parallel()
		time.Sleep(10 * time.Second)
	})

	time.Sleep(1 * time.Second)
	wg.Wait()

	t.Run("c", func(t *testing.T) {
		wg.Add(1)
		defer wg.Done()
		t.Parallel()
		time.Sleep(10 * time.Second)
	})

	t.Run("d", func(t *testing.T) {
		wg.Add(1)
		defer wg.Done()
		t.Parallel()
		time.Sleep(10 * time.Second)
	})

	wg.Wait()
}

first I ran it directly

go test parallel_test.go -v

i see the Go process was hang and no response, need ctrl+C to exit

=== RUN   TestParallelBug
=== RUN   TestParallelBug/a
=== PAUSE TestParallelBug/a
=== RUN   TestParallelBug/b
=== PAUSE TestParallelBug/b
^CFAIL  command-line-arguments  21.867s

and then I ran it by compiling it first

go test -c -o parallel.test parallel_test.go
./parallel.test -test.v

i see Go panics

=== RUN   TestParallelBug
=== RUN   TestParallelBug/a
=== PAUSE TestParallelBug/a
=== RUN   TestParallelBug/b
=== PAUSE TestParallelBug/b
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000007860, {0x112b703?, 0x246962a686ad3?}, 0x1134820)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1630 +0x405
testing.runTests.func1(0x1214e20?)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:2036 +0x45
testing.tRunner(0xc000007860, 0xc000106c88)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1576 +0x10b
testing.runTests(0xc00011a0a0?, {0x120bdb0, 0x1, 0x1}, {0x0?, 0x100c00010c8f8?, 0x0?})
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:2034 +0x489
testing.(*M).Run(0xc00011a0a0)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1906 +0x63a
main.main()
        _testmain.go:47 +0x1aa

goroutine 6 [semacquire]:
sync.runtime_Semacquire(0xc10b033f74909e88?)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/runtime/sema.go:62 +0x27
sync.(*WaitGroup).Wait(0x3b9aca00?)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/sync/waitgroup.go:116 +0x4b
command-line-arguments.TestParallelBug(0x0?)
        /Users/baozhenchen/gits/go-tester/notescripts/gotest_parallel_bug/parallel_test.go:27 +0xe5
testing.tRunner(0xc000007a00, 0x1134820)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1629 +0x3ea

goroutine 7 [chan receive]:
testing.(*T).Parallel(0xc000007d40)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1384 +0x225
command-line-arguments.TestParallelBug.func1(0x0?)
        /Users/baozhenchen/gits/go-tester/notescripts/gotest_parallel_bug/parallel_test.go:15 +0x7a
testing.tRunner(0xc000007d40, 0xc00006a630)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1629 +0x3ea

goroutine 8 [chan receive]:
testing.(*T).Parallel(0xc0001301a0)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1384 +0x225
command-line-arguments.TestParallelBug.func2(0x0?)
        /Users/baozhenchen/gits/go-tester/notescripts/gotest_parallel_bug/parallel_test.go:22 +0x7a
testing.tRunner(0xc0001301a0, 0xc00006a6a0)
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1576 +0x10b
created by testing.(*T).Run
        /Users/baozhenchen/.gvm/gos/go1.20.3/src/testing/testing.go:1629 +0x3ea

What did you expect to see?

The behaviors should be the same between running tests directly and after compiling the code first

What did you see instead?

The behaviors are different. This troubled me when running the same code between some IDEs like Goland (which will compile it first) and CI tools like Jenkins (which will run code directly)

[dmitshur]

CC @bcmills, @matloob.

[ianlancetaylor]

The difference is that when you use go test, it passes an extra argument -test.timeout=10m0s to the generated test program. This is documented as the default timeout used by go test (see the documentation of -timeout in https://pkg.go.dev/cmd/go#hdr-Testing_flags).

If you pass that same option when you run the test binary yourself, then it will also not report a deadlock. The reason there isn't a deadlock in that case is that there is a goroutine that is going to wake up in 10 minutes and say "test timed out". So the program is not deadlocked.

[rainydew]

The difference is that when you use go test, it passes an extra argument -test.timeout=10m0s to the generated test program. This is documented as the default timeout used by go test (see the documentation of -timeout in https://pkg.go.dev/cmd/go#hdr-Testing_flags).

If you pass that same option when you run the test binary yourself, then it will also not report a deadlock. The reason there isn't a deadlock in that case is that there is a goroutine that is going to wake up in 10 minutes and say "test timed out". So the program is not deadlocked.

thanks~ (I just didn't realize test binary disables the timeout totally by default)