Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: panics in example tests abort remaining tests #48009

Closed
santhosh-tekuri opened this issue Aug 27, 2021 · 2 comments
Closed

testing: panics in example tests abort remaining tests #48009

santhosh-tekuri opened this issue Aug 27, 2021 · 2 comments

Comments

@santhosh-tekuri
Copy link
Contributor

@santhosh-tekuri santhosh-tekuri commented Aug 27, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.7 darwin/amd64

Does this issue reproduce with the latest release?

not tested with go 1.17

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/santhosh/Library/Caches/go-build"
GOENV="/Users/santhosh/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/santhosh/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/santhosh/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.7"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/santhosh/contrib/golang/hello/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/xy/4qx1s2zn5qn82yzpx8xhbplr0000gn/T/go-build3446666147=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

consider a project has two example test cases as shown below:

package hello_test

import "fmt"

func ExampleOne() {
	fmt.Println("x:line1")
	panic("x:line2")
	//Output:
}

func ExampleTwo() {
	fmt.Println("y:line1")
	panic("y:line2")
	//Output:
}

now run go test and go test -v

What did you expect to see?

  1. when any example test panics, it does not verify remaining example tests
  2. go test -v does not show any fmt.Println statements in the example that before panic.
    this makes it difficult, because I can no longer use fmt.Println to debug the issue.

What did you see instead?

$ go test
--- FAIL: ExampleOne (0.00s)
panic: x:line2 [recovered]
	panic: x:line2

goroutine 1 [running]:
testing.(*InternalExample).processRunResult(0xc000073d58, 0xc00018a000, 0x8, 0x114bc, 0x300000000, 0x111af60, 0x116c650, 0x1067a68)
	/usr/local/go/src/testing/example.go:91 +0x69b
testing.runExample.func2(0xc042504ac5955240, 0xc79e6, 0x121dc20, 0xc00000e048, 0xc00000e018, 0xc0000581e0, 0xc000073d58, 0xc000073cb6, 0xc000073d88)
	/usr/local/go/src/testing/run_example.go:59 +0x11c
panic(0x111af60, 0x116c650)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/santhosh-tekuri/hello_test.ExampleOne()
	/Users/santhosh/contrib/golang/hello/example_test.go:7 +0x95
testing.runExample(0x114452e, 0xa, 0x114d0e8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/run_example.go:63 +0x222
testing.runExamples(0xc000073ed0, 0x121a1c0, 0x2, 0x2, 0xc04250e0c591f6e0)
	/usr/local/go/src/testing/example.go:44 +0x17a
testing.(*M).Run(0xc00010a000, 0x0)
	/usr/local/go/src/testing/testing.go:1418 +0x273
main.main()
	_testmain.go:45 +0x138
exit status 2
FAIL	github.com/santhosh-tekuri/hello	0.011s

notice that, in above output ExampleTwo is not shown. user may think that ExampleTwo has passed.
but actually it never ran ExampleTwo test case

$ go test -v
=== RUN   ExampleOne
--- FAIL: ExampleOne (0.00s)
panic: x:line2 [recovered]
	panic: x:line2

goroutine 1 [running]:
testing.(*InternalExample).processRunResult(0xc000073d58, 0xc00001a108, 0x8, 0x9cbb, 0x300000000, 0x111af60, 0x116c650, 0x1067a68)
	/usr/local/go/src/testing/example.go:91 +0x69b
testing.runExample.func2(0xc04250561bcd1c78, 0xcaf46, 0x121dc20, 0xc00000e048, 0xc00000e018, 0xc0000581e0, 0xc000073d58, 0xc000073cb6, 0xc000073d88)
	/usr/local/go/src/testing/run_example.go:59 +0x11c
panic(0x111af60, 0x116c650)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/santhosh-tekuri/hello_test.ExampleOne()
	/Users/santhosh/contrib/golang/hello/example_test.go:7 +0x95
testing.runExample(0x114452e, 0xa, 0x114d0e8, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/testing/run_example.go:63 +0x222
testing.runExamples(0xc000073ed0, 0x121a1c0, 0x2, 0x2, 0xc04250ec1bc8fdc8)
	/usr/local/go/src/testing/example.go:44 +0x17a
testing.(*M).Run(0xc00010a000, 0x0)
	/usr/local/go/src/testing/testing.go:1418 +0x273
main.main()
	_testmain.go:45 +0x138
exit status 2
FAIL	github.com/santhosh-tekuri/hello	0.011s

in above output i dont see fmt.Println("x:line1) output.
it means any fmt.Println before panic are lost in output.
it makes difficult to debug the issue, because developer cannot use fmt.Println to investigate why it panicked

@toothrot toothrot added this to the Backlog milestone Aug 27, 2021
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Aug 27, 2021

This is consistent with tests that use *testing.T: if a test panics, later tests are not run. I don't think we should change that lightly. Maybe it should be better documented though?

Loading

@santhosh-tekuri
Copy link
Contributor Author

@santhosh-tekuri santhosh-tekuri commented Aug 27, 2021

I agree.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants