Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add count of passing/failing tests to output #43614

Open
leighmcculloch opened this issue Jan 9, 2021 · 6 comments
Open

cmd/go: add count of passing/failing tests to output #43614

leighmcculloch opened this issue Jan 9, 2021 · 6 comments

Comments

@leighmcculloch
Copy link
Contributor

@leighmcculloch leighmcculloch commented Jan 9, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/home/leighmcculloch/local/bin"
GOCACHE="/home/leighmcculloch/.cache/go-build"
GOENV="/home/leighmcculloch/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/leighmcculloch/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/leighmcculloch/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/leighmcculloch/local/bin/go/1.16beta1"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/leighmcculloch/local/bin/go/1.16beta1/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16beta1"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/leighmcculloch/devel/gas/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3757789103=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Ran go test.

go test

What did you expect to see?

A count of how many tests failed or passed.

What did you see instead?

Just output about failed tests, and not how many failed or passed.

Feature request

When I'm iterating on a feature that is exploratory and involves breaking a lot of tests initially it is helpful to see progress as it is being made. When I program in Ruby I see a count of tests that pass/fail, and as I work I see the number of failing tests going down. More importantly when I'm tweaking code if I see that number jump up suddenly I know in the process of trying to fix things I have instantly broken many more things. This isn't a critical feature, but I think it would be helpful and useful because I've found that feature useful elsewhere.

I think it would be helpful if the go test output included at the end a count of the number of tests that:

  • executed
  • skipped
  • passed
  • failed

I have discovered that gotestsum provides visibility into number of failing tests in this form:

DONE 10 tests, 9 failures in 0.833s

Other examples:

Ruby Rspec:

Finished in 0.00068 seconds (files took 0.30099 seconds to load)
10 examples, 3 failures

Ruby minitest:

1 runs, 0 assertions, 0 failures, 1 errors, 0 skips

Python unittest:

Ran 2 tests in 0.001s

FAILED (failures=1)

Rust:

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jan 10, 2021

What consumes this output? Is it likely tools would build upon this and expect the output to be stable and machine readable?

@mvdan
Copy link
Member

@mvdan mvdan commented Jan 10, 2021

Similar to #30507.

@ALTree
Copy link
Member

@ALTree ALTree commented Jan 10, 2021

Also dup of #27755 and #41748, which was then superseded by #41878 (proposing that instead of always printing a summary, we provide an interface so that interested users can print the custom stats themselves).

@leighmcculloch
Copy link
Contributor Author

@leighmcculloch leighmcculloch commented Jan 10, 2021

What consumes this output? Is it likely tools would build upon this and expect the output to be stable and machine readable?

@davecheney I'm referring to the output of go test intended for humans, not the go test -json output intended for machines. A machine could already derive this information from the existing JSON output.


Similar to #30507.

Similar but different. That issue was concerned with an overall fail/pass indicator. This issue is concerned with test pass/fail counts.


Also dup of #27755 and #41748, which was then superseded by #41878

#41878 looks like a cool idea, but solving for a different problem to what I'm raising here. It provides functionality for customization, I don't think the problem in this issue needs customization. #41748 is also a cool idea, but is proposing printing test names at the end, and not a count summary.


#27755 looks to be almost what I'm proposing. That issue was closed and locked with little discussion.

In that issue @rsc states:

If you want to build alternate displays, the new -json flag lets programs consume the test results and customize the output arbitrarily. That's probably the right thing to do instead of changing the default display.

What I'm suggesting is not functionality to build alternate displays but additional log information that exposes information that is commonly included in the test output of similar tools into the default output. I think it is worth reconsidering that, either as #27755 proposed with counts per package, or more simply a total summary at the end, which is much more common and like how other languages test tools function.

@cagedmantis cagedmantis added this to the Backlog milestone Jan 15, 2021
@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jan 15, 2021

@dnephin
Copy link
Contributor

@dnephin dnephin commented Apr 11, 2021

gotestsum provides this summary, for example:

DONE 101 tests[, 3 skipped][, 2 failures][, 1 error] in 0.103s

It seems like a decision was already made on this issue in #27755. gotestsum is one example of the recommended solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants