cmd/go: add count of passing/failing tests to output

[leighmcculloch]

What version of Go are you using (go version)?

$ go version
go version go1.16beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/leighmcculloch/local/bin"
GOCACHE="/home/leighmcculloch/.cache/go-build"
GOENV="/home/leighmcculloch/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/leighmcculloch/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/leighmcculloch/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/leighmcculloch/local/bin/go/1.16beta1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/leighmcculloch/local/bin/go/1.16beta1/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16beta1"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/leighmcculloch/devel/gas/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3757789103=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran go test.

go test

What did you expect to see?

A count of how many tests failed or passed.

What did you see instead?

Just output about failed tests, and not how many failed or passed.

Feature request

When I'm iterating on a feature that is exploratory and involves breaking a lot of tests initially it is helpful to see progress as it is being made. When I program in Ruby I see a count of tests that pass/fail, and as I work I see the number of failing tests going down. More importantly when I'm tweaking code if I see that number jump up suddenly I know in the process of trying to fix things I have instantly broken many more things. This isn't a critical feature, but I think it would be helpful and useful because I've found that feature useful elsewhere.

I think it would be helpful if the go test output included at the end a count of the number of tests that:

• executed
• skipped
• passed
• failed

I have discovered that gotestsum provides visibility into number of failing tests in this form:

DONE 10 tests, 9 failures in 0.833s

Other examples:

Ruby Rspec:

Finished in 0.00068 seconds (files took 0.30099 seconds to load)
10 examples, 3 failures

Ruby minitest:

1 runs, 0 assertions, 0 failures, 1 errors, 0 skips

Python unittest:

Ran 2 tests in 0.001s

FAILED (failures=1)

Rust:

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

[davecheney]

What consumes this output? Is it likely tools would build upon this and expect the output to be stable and machine readable?

[mvdan]

Similar to #30507.

[ALTree]

Also dup of #27755 and #41748, which was then superseded by #41878 (proposing that instead of always printing a summary, we provide an interface so that interested users can print the custom stats themselves).

[leighmcculloch]

What consumes this output? Is it likely tools would build upon this and expect the output to be stable and machine readable?

@davecheney I'm referring to the output of go test intended for humans, not the go test -json output intended for machines. A machine could already derive this information from the existing JSON output.

---

Similar to #30507.

Similar but different. That issue was concerned with an overall fail/pass indicator. This issue is concerned with test pass/fail counts.

---

Also dup of #27755 and #41748, which was then superseded by #41878

#41878 looks like a cool idea, but solving for a different problem to what I'm raising here. It provides functionality for customization, I don't think the problem in this issue needs customization. #41748 is also a cool idea, but is proposing printing test names at the end, and not a count summary.

---

#27755 looks to be almost what I'm proposing. That issue was closed and locked with little discussion.

In that issue @rsc states:

If you want to build alternate displays, the new -json flag lets programs consume the test results and customize the output arbitrarily. That's probably the right thing to do instead of changing the default display.

What I'm suggesting is not functionality to build alternate displays but additional log information that exposes information that is commonly included in the test output of similar tools into the default output. I think it is worth reconsidering that, either as #27755 proposed with counts per package, or more simply a total summary at the end, which is much more common and like how other languages test tools function.

[cagedmantis]

/cc @bcmills @jayconrod @matloob

[dnephin]

gotestsum provides this summary, for example:

DONE 101 tests[, 3 skipped][, 2 failures][, 1 error] in 0.103s

It seems like a decision was already made on this issue in #27755. gotestsum is one example of the recommended solution.

[TheOnlyWei]

Would be interested in a summary of runs, passes, and failures as well. We have a project with hundreds of tests and need some form of test result summary. This is for reporting to higher-ups about the state of our products or reporting test coverages at a glance. Currently attempting to aggregate the lines of output from the test runs using PowerShell Core with the following code (Select-String is similar to grep):

$runs = ($testOutput | Select-String -Pattern @("=== RUN") -NoEmphasis).Count
$passes = ($testOutput | Select-String -Pattern @("--- PASS") -NoEmphasis).Count
$failures = ($testOutput | Select-String -Pattern @("--- FAIL") -NoEmphasis).Count

From the above, $runs != $passes + $failures. So aggregation by user is difficult as we don't know how Golang decides to output these lines.

[MichaelSnowden]

If you're just looking for a quick workaround to find out the pass rate of a test like I was, you can use jq like this:

go test <whatever> -count <n> -json | jq -s 'map(select(.Test and .Action == "pass")) | length'