testing: panics in example tests abort remaining tests

[santhosh-tekuri]

What version of Go are you using (go version)?

$ go version
go version go1.16.7 darwin/amd64

Does this issue reproduce with the latest release?

not tested with go 1.17

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/santhosh/Library/Caches/go-build"
GOENV="/Users/santhosh/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/santhosh/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/santhosh/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.7"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/santhosh/contrib/golang/hello/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/xy/4qx1s2zn5qn82yzpx8xhbplr0000gn/T/go-build3446666147=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

consider a project has two example test cases as shown below:

package hello_test

import "fmt"

func ExampleOne() {
	fmt.Println("x:line1")
	panic("x:line2")
	//Output:
}

func ExampleTwo() {
	fmt.Println("y:line1")
	panic("y:line2")
	//Output:
}

now run go test and go test -v

What did you expect to see?

1. when any example test panics, it does not verify remaining example tests
2. go test -v does not show any fmt.Println statements in the example that before panic.
   this makes it difficult, because I can no longer use fmt.Println to debug the issue.

What did you see instead?

$ go test
--- FAIL: ExampleOne (0.00s)
panic: x:line2 [recovered]
	panic: x:line2

goroutine 1 [running]:
testing.(*InternalExample).processRunResult(0xc000073d58, 0xc00018a000, 0x8, 0x114bc, 0x300000000, 0x111af60, 0x116c650, 0x1067a68)
	/usr/local/go/src/testing/example.go:91 +0x69b
testing.runExample.func2(0xc042504ac5955240, 0xc79e6, 0x121dc20, 0xc00000e048, 0xc00000e018, 0xc0000581e0, 0xc000073d58, 0xc000073cb6, 0xc000073d88)
	/usr/local/go/src/testing/run_example.go:59 +0x11c
panic(0x111af60, 0x116c650)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/santhosh-tekuri/hello_test.ExampleOne()
	/Users/santhosh/contrib/golang/hello/example_test.go:7 +0x95
testing.runExample(0x114452e, 0xa, 0x114d0e8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/run_example.go:63 +0x222
testing.runExamples(0xc000073ed0, 0x121a1c0, 0x2, 0x2, 0xc04250e0c591f6e0)
	/usr/local/go/src/testing/example.go:44 +0x17a
testing.(*M).Run(0xc00010a000, 0x0)
	/usr/local/go/src/testing/testing.go:1418 +0x273
main.main()
	_testmain.go:45 +0x138
exit status 2
FAIL	github.com/santhosh-tekuri/hello	0.011s

notice that, in above output ExampleTwo is not shown. user may think that ExampleTwo has passed.
but actually it never ran ExampleTwo test case

$ go test -v
=== RUN   ExampleOne
--- FAIL: ExampleOne (0.00s)
panic: x:line2 [recovered]
	panic: x:line2

goroutine 1 [running]:
testing.(*InternalExample).processRunResult(0xc000073d58, 0xc00001a108, 0x8, 0x9cbb, 0x300000000, 0x111af60, 0x116c650, 0x1067a68)
	/usr/local/go/src/testing/example.go:91 +0x69b
testing.runExample.func2(0xc04250561bcd1c78, 0xcaf46, 0x121dc20, 0xc00000e048, 0xc00000e018, 0xc0000581e0, 0xc000073d58, 0xc000073cb6, 0xc000073d88)
	/usr/local/go/src/testing/run_example.go:59 +0x11c
panic(0x111af60, 0x116c650)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/santhosh-tekuri/hello_test.ExampleOne()
	/Users/santhosh/contrib/golang/hello/example_test.go:7 +0x95
testing.runExample(0x114452e, 0xa, 0x114d0e8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/run_example.go:63 +0x222
testing.runExamples(0xc000073ed0, 0x121a1c0, 0x2, 0x2, 0xc04250ec1bc8fdc8)
	/usr/local/go/src/testing/example.go:44 +0x17a
testing.(*M).Run(0xc00010a000, 0x0)
	/usr/local/go/src/testing/testing.go:1418 +0x273
main.main()
	_testmain.go:45 +0x138
exit status 2
FAIL	github.com/santhosh-tekuri/hello	0.011s

in above output i dont see fmt.Println("x:line1) output.
it means any fmt.Println before panic are lost in output.
it makes difficult to debug the issue, because developer cannot use fmt.Println to investigate why it panicked

[jayconrod]

This is consistent with tests that use *testing.T: if a test panics, later tests are not run. I don't think we should change that lightly. Maybe it should be better documented though?

[santhosh-tekuri]

I agree.