New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: document exit codes of a process executing `go test` #25989

Open
kostix opened this Issue Jun 21, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@kostix

kostix commented Jun 21, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10.3

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/devel/go"
GORACE=""
GOROOT="/home/user/devel/golang1.10"
GOTMPDIR=""
GOTOOLDIR="/home/user/devel/golang1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build141686204=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I run go help test and try to search for keywords "return", "exit", "code" in the output.

What did you expect to see?

I expect the possible exit codes for running go test be documented
in order to make sensible decisions about running this command in scripts
(and from Makefiles, by a CI system etc).

What did you see instead?

No menton of the return/exit code.


I'd expect the following cases to be documented if implemented:

  • Successful run—all tests compiled and passed OK—exit code 0.
  • Failed run—the compilation of at least one package failed or at least a single test failed—a non-zero exit code.
  • Invalid parameters passed and/or one of the listed packages wasn't found—a non-zero exit code,
    different from that of the former case.

I'd also think that the cached results should effectively "memorize" the exit codeused ATM the results were generated (and cached).

See also #23716.

@GTB3NW

This comment has been minimized.

GTB3NW commented Jun 21, 2018

Ignore me I'm an idiot. t.Error != t.Fail

I agree with the OP however, it should have a none 0 status, 1 for error, 2 for fail maybe?

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jun 21, 2018

Right now the exit status is 1 for most failures, 2 if the build was interrupted.

@smasher164

This comment has been minimized.

Contributor

smasher164 commented Jun 22, 2018

Do we distinguish between flag parsing errors where different exit codes are output? For example, passing in "-c=invalid" returns a 2

$ go test -c=invalid
go test: illegal bool flag value invalid
run "go help test" or "go help testflag" for more information
$ echo $?
2

while passing in "-covermode=invalid" returns a 1.

$ go test -covermode=invalid
invalid flag argument for -covermode: "invalid"
$ echo $?
1
@kostix

This comment has been minimized.

kostix commented Jun 22, 2018

@GTB3NW note that your particular problem is most possibly not related to the essense of this issue which is about documenting the implemented behaviour which appears to be OK.
Here at my $dayjob, in our CI pipeline, we call go test multiple times with different command-line options (say, including -race for nightly tests and also -bench … etc), and go test has always exited with a non-zero exit code on error since, I reckon, at least v1.6, and I really have no reason to think it was otherwise before.

I've created this issue merely because such things ought to be documented for reference purposes (say, for DevOps folks which may be not too familiar with Go to implement the CI pipeline).

Try to figure out what may mask the exit code of your go test process.
Note that all /bin/sh-style shells found in commodity OSes by default run in a mode which ignores any errors, that means, if you have a script like

go test ...
echo Whatever

this code won't ever exit if go test fails.
(And these shells did this since forever; and this is even mandated by POSIX.)
You either need to re-write your scripts so that they run with set -e -u (almost always a good thing to do) or explicitly test whether a command of interest failed, like in

go test ...
rc=$?
if [ $rc -ne 0 ]; then
  echo "testing failed" >&2
  exit $rc
fi

Updated @GTB3NW, ah, I see you got sorted it out :-)
Sorry for the noise then—should have re-read the thread rather than relying on comment e-mailing (which ignores edits).

@GTB3NW

This comment has been minimized.

GTB3NW commented Jun 22, 2018

Okay here's one for you then, I've read the documentation, perhaps I'm missing something here, but the following should FAIL and return error code 1 should it not?

blahtest/blah_test.go:

package blahtest

import "testing"

func TestBlah(t *testing.T) {
	t.Fatal("Blah")
}
go test <redacted>/blahtest -v

=== RUN   TestBlah
--- FAIL: TestBlah (0.00s)
        blah_test.go:6: Blah
FAIL
ok      <redacted>/blahtest      0.004s

Should that not be a FAIL on /blahtest too?

The aim is to just run:

go test -v -short $(go list ${PKG}./... | grep -v /vendor/)

Test all the packages and if any one of them fails the whole test fails, surely that's a common requirement. Either I'm misunderstanding the testing facilities/there's a bug/go test is lacking something critical.

Could anyone set me in the right direction please I'd really appreciate it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment