Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: fail tests that invoke os.Exit(0) explicitly #29062

Closed
rhysd opened this issue Dec 2, 2018 · 45 comments
Closed

cmd/go: fail tests that invoke os.Exit(0) explicitly #29062

rhysd opened this issue Dec 2, 2018 · 45 comments

Comments

@rhysd
Copy link

@rhysd rhysd commented Dec 2, 2018

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rhysd/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rhysd/.go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9t/jwm1hlr905g_wlnzrmbnb3cr0000gn/T/go-build385579133=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Here is a code snip to explain this issue:

package foo

import (
	"os"
	"testing"
)

type Parsed struct{}

func parseArguments(args []string) (Parsed, error) {
	// parse arguments
	if len(args) == 0 {
		// Show help message
		os.Exit(0)
	}

	// check parsed arguments

	return Parsed{}, nil
}

func TestParse(t *testing.T) {
	_, err := parseArguments([]string{})
	if err != nil {
		t.Fatal(err)
	}
	// test parse result
}

func TestOther(t *testing.T) {
	t.Fatal()
}

Please write above code to some Go file and run:

$ go test
$ echo $?

It outputs 0. So it means that test is ok. However, actually test has stopped at the middle of execution since os.Exit(0) is accidentally called.
I'm not sure that this is a bug. It may be intended behavior. But when calling os.Exit(0) in tests accidentally (for example, due to lack of understanding of API), I may not notice the tests are wrongly run since it exits successfully. CI also cannot detect it.

What did you expect to see?

IMO, go test exiting with non-zero exit status when the tests exit at the middle of execution by os.Exit() would solve this issue.

What did you see instead?

echo $? echoes 0 and test exited successfully

@rhysd rhysd changed the title 'go test' successfully exits even if the program calls os.Exit(0) 'go test' successfully exits even if some test calls os.Exit(0) Dec 2, 2018
@mvdan mvdan added the NeedsDecision label Dec 2, 2018
@mvdan mvdan changed the title 'go test' successfully exits even if some test calls os.Exit(0) cmd/go: test seems to succeed even if some test calls os.Exit(0) Dec 2, 2018
@mvdan
Copy link
Member

@mvdan mvdan commented Dec 2, 2018

Here is a smaller repro:

$ cat go.mod
module foo.bar
$ cat f_test.go
package foo

import (
        "os"
        "testing"
)

func TestExit(t *testing.T) {
        os.Exit(0)
}

func TestFatal(t *testing.T) {
        t.Fatal()
}
$ go test
ok      foo.bar 0.001s
$ go test -v
=== RUN   TestExit
ok      foo.bar 0.001s

I agree that this can be confusing, since go test appears happy. It's only go test -v that looks off, since none of the tests actually succeed and finish.

Perhaps go test could error if a test binary exited too soon. For example, if we do go test -c and run the test binary, we get no output at all instead of a PASS or a FAIL. But I'm not sure if we could implement that without introducing even more confusing edge cases. For example, what should happen if a test binary panics?

/cc @bcmills @ianlancetaylor

@mvdan
Copy link
Member

@mvdan mvdan commented Dec 2, 2018

A perhaps simple fix would be for go test to complain if a test binary prints absolutely nothing. I think that should never happen.

@randall77
Copy link
Contributor

@randall77 randall77 commented Dec 4, 2018

@mvdan: That fix sounds like the right one.

@bcmills bcmills added this to the Go1.13 milestone Dec 4, 2018
@bcmills
Copy link
Member

@bcmills bcmills commented Dec 4, 2018

A test binary can provide an arbitrary func main, right? Why does it need to print anything on success?

@randall77
Copy link
Contributor

@randall77 randall77 commented Dec 4, 2018

Are you talking about TestMain? It prints a result (PASS) as well.
I suppose you could implement TestMain without calling m.Run on its argument, but that seems not supported.

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 5, 2018

I did some experimenting and it looks like I was just mistaken. Carry on!

@bcmills bcmills changed the title cmd/go: test seems to succeed even if some test calls os.Exit(0) cmd/go: fail tests that invoke os.Exit(0) explicitly Jan 17, 2019
@mvdan mvdan self-assigned this Jul 1, 2019
@mvdan mvdan added NeedsFix and removed NeedsDecision labels Jul 1, 2019
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 1, 2019

Change https://golang.org/cl/184457 mentions this issue: cmd/go: fail if a test binary succeeds with no output

@eliben
Copy link
Member

@eliben eliben commented Aug 31, 2019

From the CL:

A few TestMain funcs in the standard library needed fixing, as they
returned without printing anything as a means to skip testing the entire
package

How concerning is this for breaking user tests? If you had to fix up stdlib tests, it's likely that many tests in user code are going to break.

@randall77 #31969 mentions a TestMain that doesn't invoke m.Run (though the docs of testing ask not to do that). Also, as @mvdan's CL demonstrates, this is done in the stdlib: https://github.com/golang/go/blob/master/src/cmd/internal/goobj/goobj_test.go#L33

@mvdan
Copy link
Member

@mvdan mvdan commented Sep 9, 2019

Historically, it has been acceptable to break user programs that were doing the opposite of what is documented. For example, see #31859. We want to avoid this breakage whenever possible, but in some cases it's the best option.

I think this is one of those cases. The fix will uncover package tests that were succeeding by accident, which was the motivation of this bug. And after all, fixing the broken tests was a two-line change, and I think that's a reasonable thing to ask of the users who weren't following the docs.

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 9, 2019

Change https://golang.org/cl/200104 mentions this issue: doc/go1.14: note that tests without any output now fail

@gopherbot
Copy link

@gopherbot gopherbot commented Oct 9, 2019

Change https://golang.org/cl/200106 mentions this issue: Revert "cmd/go: fail if a test binary exits with no output"

@bcmills bcmills reopened this Oct 9, 2019
@bcmills
Copy link
Member

@bcmills bcmills commented Oct 9, 2019

I'm reverting this CL due to #34791. We can un-revert it if we decide that it is acceptable to break the assumptions of #18153, or if we figure out a way to implement this change without breaking those assumptions.

gopherbot pushed a commit that referenced this issue Oct 9, 2019
This reverts CL 184457.

Reason for revert: introduced failures in the regression test for #18153.

Fixes #34791
Updates #29062

Change-Id: I4040965163f809083c023be055e69b1149d6214e
Reviewed-on: https://go-review.googlesource.com/c/go/+/200106
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@mvdan
Copy link
Member

@mvdan mvdan commented Oct 10, 2019

Thanks for taking care of this @bcmills. I do fundamentally disagree with the premise of #18153. For example, it makes tests brittle, and requires them to be run with a real terminal, and no arguments:

$ cat f_test.go
package foo

import (
        "os"
        "testing"

        "golang.org/x/crypto/ssh/terminal"
)

func TestFoo(t *testing.T) {
        if !terminal.IsTerminal(int(os.Stdout.Fd())) {
                t.Fatal("not a terminal!")
        }
}
$ go test
PASS
ok      test.tld/foo    0.002s
$ go test | cat
--- FAIL: TestFoo (0.00s)
    f_test.go:12: not a terminal!
FAIL
exit status 1
FAIL    test.tld/foo    0.002s
$ go test .
--- FAIL: TestFoo (0.00s)
    f_test.go:12: not a terminal!
FAIL
FAIL    test.tld/foo    0.002s
FAIL

go test -v with no arguments fails, too.

Moreover, it makes it very hard (or impossible, I suspect) to inspect or modify the output in any way, like we needed to do here.

Do you reckon a proposal is necessary to make a proper decision here?

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 10, 2019

I realise that the original idea was to color test output, and not necessarily make tests fail or pass depending on stdout being a terminal. Still, I think it's wrong to change the test's behavior depending on whether it happens to be run on a terminal. If a user really wants color on their test outputs, they could use a test flag, or an environment variable like $TERM != "dumb". At least in that case, you wouldn't get the inconsistencies I showed above.

@bcmills
Copy link
Member

@bcmills bcmills commented Dec 4, 2019

no one reinvokes the test binary with all the original test flags so the flag gets cleared "for free"

I have a Google-internal exectest package (a helper library for exec-based tests) that does preserve the original test flags — it filters out only -test.run.

That said, it also does a lot of other non-portable things (like introspecting the program counter to figure out which function calls to skip or replay in the subprocess), so I would not be at all upset to have to filter out an additional flag.

@rsc rsc moved this from Active to Likely Accept in Proposals Dec 4, 2019
@rsc
Copy link
Contributor

@rsc rsc commented Dec 11, 2019

No change in consensus. Accepted.

@rsc rsc changed the title proposal: cmd/go: fail tests that invoke os.Exit(0) explicitly cmd/go: fail tests that invoke os.Exit(0) explicitly Dec 11, 2019
@rsc rsc modified the milestones: Proposal, Go1.15 Dec 11, 2019
@rsc rsc moved this from Likely Accept to Accepted in Proposals Dec 11, 2019
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Feb 19, 2020

This issue is currently labeled as early-in-cycle for Go 1.15. That time is now, so friendly ping. If it no longer needs to be done early in cycle, that label can be removed.

@gopherbot
Copy link

@gopherbot gopherbot commented Mar 5, 2020

Change https://golang.org/cl/222243 mentions this issue: cmd/go: make go test -json report failures for panicking/exiting tests

gopherbot pushed a commit that referenced this issue Mar 6, 2020
'go test -json' should report that a test failed if the test binary
did not exit normally with status 0. This covers panics, non-zero
exits, and abnormal terminations.

These tests don't print a final result when run with -test.v (which is
used by 'go test -json'). The final result should be "PASS" or "FAIL"
on a line by itself. 'go test' prints "FAIL" in this case, but
includes error information.

test2json was changed in CL 192104 to report that a test passed if it
does not report a final status. This caused 'go test -json' to report
that a test passed after a panic or non-zero exit.

With this change, test2json treats "FAIL" with error information the
same as "FAIL" on a line by itself. This is intended to be a minimal
fix for backporting, but it will likely be replaced by a complete
solution for #29062.

Fixes #37555
Updates #29062
Updates #31969

Change-Id: Icb67bcd36bed97e6a8d51f4d14bf71f73c83ac3d
Reviewed-on: https://go-review.googlesource.com/c/go/+/222243
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Mar 9, 2020

Change https://golang.org/cl/222658 mentions this issue: [release-branch.go1.14] cmd/go: make go test -json report failures for panicking/exiting tests

gopherbot pushed a commit that referenced this issue Mar 9, 2020
…r panicking/exiting tests

'go test -json' should report that a test failed if the test binary
did not exit normally with status 0. This covers panics, non-zero
exits, and abnormal terminations.

These tests don't print a final result when run with -test.v (which is
used by 'go test -json'). The final result should be "PASS" or "FAIL"
on a line by itself. 'go test' prints "FAIL" in this case, but
includes error information.

test2json was changed in CL 192104 to report that a test passed if it
does not report a final status. This caused 'go test -json' to report
that a test passed after a panic or non-zero exit.

With this change, test2json treats "FAIL" with error information the
same as "FAIL" on a line by itself. This is intended to be a minimal
fix for backporting, but it will likely be replaced by a complete
solution for #29062.

Fixes #37671
Updates #37555
Updates #29062
Updates #31969

Change-Id: Icb67bcd36bed97e6a8d51f4d14bf71f73c83ac3d
Reviewed-on: https://go-review.googlesource.com/c/go/+/222243
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
(cherry picked from commit 5ea58c6)
Reviewed-on: https://go-review.googlesource.com/c/go/+/222658
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
@jayconrod jayconrod modified the milestones: Go1.15, Go1.16 Apr 29, 2020
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 12, 2020

This issue is currently labeled as early-in-cycle for Go 1.16.
That time is now, so this is a friendly ping so the issue is looked at again.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 27, 2020

Change https://golang.org/cl/250977 mentions this issue: cmd/go, testing, os: fail test that calls os.Exit(0)

@gopherbot gopherbot closed this in 4f76fe8 Aug 27, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 28, 2020

Change https://golang.org/cl/251262 mentions this issue: testing: restore os.Exit(0) after every call to (*M).Run

gopherbot pushed a commit that referenced this issue Aug 28, 2020
cmd/go.TestScript/test_main_twice demonstrates a program that invokes
(*M).Run twice in a row. If we only restore os.Exit(0) in m.afterOnce,
we will fail to restore it after the second run and fail the test
process despite both runs passing.

Updates #29062
Updates #23129

Change-Id: Id22ec68f1708e4583c8dda14a8ba0efae7178b85
Reviewed-on: https://go-review.googlesource.com/c/go/+/251262
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jan 14, 2021

Change https://golang.org/cl/283873 mentions this issue: cmd/test2json: document passing -test.paniconexit0

gopherbot pushed a commit that referenced this issue Jan 14, 2021
For #29062
Fixes #43263

Change-Id: I160197c94cc4f936967cc22c82cec01663a14fe6
Reviewed-on: https://go-review.googlesource.com/c/go/+/283873
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Accepted
Linked pull requests

Successfully merging a pull request may close this issue.

None yet