testing: silly typo in subtests causes hard-to-understand error message

[barnex]

What did you do?

Run go test on this (wrong) test file. The subtest is wrongfully referring to the outer testing.T:

package main

import "testing"

func TestPanic(t *testing.T) {
        t.Run("", func(typo *testing.T) {
                t.Fatalf("...") // wrong, should be typo.Fatalf, not t.Fatalf
        })
}

What did you expect to see?

Perhaps a somewhat informative error message. At least not a panic(nil) leading to a long debug session.

What did you see instead?

A very confusing panic(nil):

> go test
panic: test executed panic(nil) or runtime.Goexit

goroutine 6 [running]:
testing.tRunner.func1(0xc4200941e0)
        /home/arne/bin/go/src/testing/testing.go:712 +0x2d2
runtime.Goexit()
        /home/arne/bin/go/src/runtime/panic.go:366 +0x147
testing.(*common).FailNow(0xc4200940f0)
        /home/arne/bin/go/src/testing/testing.go:544 +0x39
testing.(*common).Fatalf(0xc4200940f0, 0x53665f, 0xd, 0x0, 0x0, 0x0)
        /home/arne/bin/go/src/testing/testing.go:588 +0x83
_/home/arne/tmp.TestPanic.func1(0xc4200941e0)
        /home/arne/tmp/panic_test.go:7 +0x5c
testing.tRunner(0xc4200941e0, 0xc42000e400)
        /home/arne/bin/go/src/testing/testing.go:747 +0xd0
created by testing.(*T).Run
        /home/arne/bin/go/src/testing/testing.go:790 +0x2de
exit status 2
FAIL    _/home/arne/tmp 0.003s

Does this issue reproduce with the latest release (go1.8.3)?

Yes. Also with go1.9rc1.

System details

go version go1.9rc1 linux/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/arne"
GORACE=""
GOROOT="/home/arne/bin/go"
GOTOOLDIR="/home/arne/bin/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build514931481=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOROOT/bin/go version: go version go1.9rc1 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.9rc1
uname -sr: Linux 4.4.0-83-generic
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial
/lib/x86_64-linux-gnu/libc.so.6: GNU C Library (Ubuntu GLIBC 2.23-0ubuntu9) stable release version 2.23, by Roland McGrath et al.
gdb --version: GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

[ALTree]

What did you see instead?
A very confusing panic(nil)

What did you expect to see?
At least not a panic(nil)

The full error message is not

test executed panic(nil)

it's

test executed panic(nil) or runtime.Goexit

the part after the or is the relevant one.

Your erroneous Fatalf call causes a FailNow call, which triggers a runtime.Goexit on the external goroutine. The error message is (correctly) informing you of this (the fact that runtime.Goexit was called on the external goroutine before it was done with its job).

What's really misleading here is the fact that the error message refers to a runtime.Goexit call, which the user has probably no way to trace back to t.FailNow without reading the testing.go source code.

Suggestion: amend the FailNow doc by mentioning the fact that it calls runtime.Goexit, i.e. from this:

FailNow marks the function as having failed and stops its execution.

to this:

FailNow marks the function as having failed and stops its execution by calling runtime.Goexit.

This way a user reading about a crash caused by runtime.Goexit will have a way to trace the issue back to a bad t.FailNow call without having to read the testing package source code.

[ALTree]

Oh, and the SkipNow doc, too. It calls runtime.Goexit, like FailNow.

[cznic]

Related #17421.

[gopherbot]

Change https://golang.org/cl/52770 mentions this issue: testing: explain how SkipNow and FailNow stop execution