New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: test seems to succeed even if some test calls os.Exit(0) #29062

Open
rhysd opened this Issue Dec 2, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@rhysd

rhysd commented Dec 2, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rhysd/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rhysd/.go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9t/jwm1hlr905g_wlnzrmbnb3cr0000gn/T/go-build385579133=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Here is a code snip to explain this issue:

package foo

import (
	"os"
	"testing"
)

type Parsed struct{}

func parseArguments(args []string) (Parsed, error) {
	// parse arguments
	if len(args) == 0 {
		// Show help message
		os.Exit(0)
	}

	// check parsed arguments

	return Parsed{}, nil
}

func TestParse(t *testing.T) {
	_, err := parseArguments([]string{})
	if err != nil {
		t.Fatal(err)
	}
	// test parse result
}

func TestOther(t *testing.T) {
	t.Fatal()
}

Please write above code to some Go file and run:

$ go test
$ echo $?

It outputs 0. So it means that test is ok. However, actually test has stopped at the middle of execution since os.Exit(0) is accidentally called.
I'm not sure that this is a bug. It may be intended behavior. But when calling os.Exit(0) in tests accidentally (for example, due to lack of understanding of API), I may not notice the tests are wrongly run since it exits successfully. CI also cannot detect it.

What did you expect to see?

IMO, go test exiting with non-zero exit status when the tests exit at the middle of execution by os.Exit() would solve this issue.

What did you see instead?

echo $? echoes 0 and test exited successfully

@rhysd rhysd changed the title from 'go test' successfully exits even if the program calls os.Exit(0) to 'go test' successfully exits even if some test calls os.Exit(0) Dec 2, 2018

@mvdan mvdan added the NeedsDecision label Dec 2, 2018

@mvdan mvdan changed the title from 'go test' successfully exits even if some test calls os.Exit(0) to cmd/go: test seems to succeed even if some test calls os.Exit(0) Dec 2, 2018

@mvdan

This comment has been minimized.

Member

mvdan commented Dec 2, 2018

Here is a smaller repro:

$ cat go.mod
module foo.bar
$ cat f_test.go
package foo

import (
        "os"
        "testing"
)

func TestExit(t *testing.T) {
        os.Exit(0)
}

func TestFatal(t *testing.T) {
        t.Fatal()
}
$ go test
ok      foo.bar 0.001s
$ go test -v
=== RUN   TestExit
ok      foo.bar 0.001s

I agree that this can be confusing, since go test appears happy. It's only go test -v that looks off, since none of the tests actually succeed and finish.

Perhaps go test could error if a test binary exited too soon. For example, if we do go test -c and run the test binary, we get no output at all instead of a PASS or a FAIL. But I'm not sure if we could implement that without introducing even more confusing edge cases. For example, what should happen if a test binary panics?

/cc @bcmills @ianlancetaylor

@mvdan

This comment has been minimized.

Member

mvdan commented Dec 2, 2018

A perhaps simple fix would be for go test to complain if a test binary prints absolutely nothing. I think that should never happen.

@randall77

This comment has been minimized.

Contributor

randall77 commented Dec 4, 2018

@mvdan: That fix sounds like the right one.

@bcmills bcmills added this to the Go1.13 milestone Dec 4, 2018

@bcmills

This comment has been minimized.

Member

bcmills commented Dec 4, 2018

A test binary can provide an arbitrary func main, right? Why does it need to print anything on success?

@randall77

This comment has been minimized.

Contributor

randall77 commented Dec 4, 2018

Are you talking about TestMain? It prints a result (PASS) as well.
I suppose you could implement TestMain without calling m.Run on its argument, but that seems not supported.

@bcmills

This comment has been minimized.

Member

bcmills commented Dec 5, 2018

I did some experimenting and it looks like I was just mistaken. Carry on!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment