Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: add summary at end of go test results #30507

Open
markvincentcaro opened this Issue Mar 1, 2019 · 11 comments

Comments

Projects
None yet
5 participants
@markvincentcaro
Copy link

markvincentcaro commented Mar 1, 2019

Currently, there is no way to quickly tell if a test failure occurred after running go test.

You have to scroll up and look for the word FAIL or search for it in a text editor.

Printing a summary of the test results at the bottom (e.g. Tests run: 10, Failures: 2, Errors: 0, Skipped: 0) would make it easy to tell at a glance if any test failures occurred or not.

An answer to Issue #27755 suggested using the -json flag, but I haven't seen any good documentation or examples on how to use it. Also, this approach seems too convoluted for something that other languages have for tests out of the box.

EDIT: I'm on Windows 10.

@gopherbot gopherbot added this to the Proposal milestone Mar 1, 2019

@gopherbot gopherbot added the Proposal label Mar 1, 2019

@markvincentcaro markvincentcaro changed the title Proposal: cmd/go test - Add Test Summary at End of Results proposal: cmd/go test - Add Test Summary at End of Results Mar 1, 2019

@markvincentcaro markvincentcaro changed the title proposal: cmd/go test - Add Test Summary at End of Results proposal: cmd/go test - Add Summary at End of Test Results Mar 1, 2019

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Mar 1, 2019

Currently, there is no way to quickly tell if a test failure occurred after running go test

The very last line should tell you. Even if you're testing many packages, each package that succeeded should only output a single line, so I don't see how this would be a problem unless you were testing a thousand packages at once.

Or, if you simply want to tell if any test failed, just check go test's exit code.

@markvincentcaro

This comment has been minimized.

Copy link
Author

markvincentcaro commented Mar 1, 2019

@mvdan

Even if you're testing many packages, each package that succeeded should only output a single line, so I don't see how this would be a problem unless you were testing a thousand packages at once.

Actually, the fact that you need to manually check if any package failed can be subject to human errors, however rare they may be. And you shouldn't be discouraged from decoupling your code into organized packages and writing tests for them just because you want to keep your package count low to make test verification easier. The compiler's tools should be doing that for you.

Or, if you simply want to tell if any test failed, just check go test's exit code.

How exactly do you do this?

@ianlancetaylor ianlancetaylor changed the title proposal: cmd/go test - Add Summary at End of Test Results proposal: cmd/go: add summary at end of go test results Mar 1, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Mar 1, 2019

Or, if you simply want to tell if any test failed, just check go test's exit code.

How exactly do you do this?

What system are you running on? GNU/Linux? Windows?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

ianlancetaylor commented Mar 1, 2019

Programmatically you check the exit code by checking whether os/exec's Run method return an error or not.

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Mar 1, 2019

On unix-y shells, you can do go test whatever-packages; echo $? to see the last command's exit status. I'm sure Windows and all other systems will have similar mechanisms.

@mvdan mvdan added the WaitingForInfo label Mar 1, 2019

@markvincentcaro

This comment has been minimized.

Copy link
Author

markvincentcaro commented Mar 1, 2019

@mvdan
What info do you need? If it's my system, I added it in the description.

@mvdan

This comment has been minimized.

Copy link
Member

mvdan commented Mar 1, 2019

Well, @ianlancetaylor asked a question, and we both showed that it's generally simple to check an exit status code. You were asking how to quickly tell if a go test run had any failures - that's the simplest way to check. We're waiting for info because your entire proposal seems to be based around that problem.

@markvincentcaro

This comment has been minimized.

Copy link
Author

markvincentcaro commented Mar 2, 2019

@mvdan

The answer from this SO thread (https://stackoverflow.com/a/334890/7824245) says the errorlevel environment variable is how to check it on Windows, but to be safe, it also needs set errorlevel= to clear any previous values.

I'll try it out and see how that works.

@markvincentcaro

This comment has been minimized.

Copy link
Author

markvincentcaro commented Mar 5, 2019

@mvdan @ianlancetaylor

Nevermind, I found a better way to quickly detect test failures from this SO thread:

go test ./... -v -short > test-results.txt & findstr "FAIL _test" test-results.txt

Though I still kinda wish that the Go team would officially add this feature in the future.

@rsc

This comment has been minimized.

Copy link
Contributor

rsc commented Mar 7, 2019

Note that #27755 was about a per-package summary. This issue is about a per-run summary over many packages. In general I'm skeptical of summaries: totaling numbers across tests is misleading in various ways, and people won't agree about the level of detail. But I do sympathize with seeing the end of output and not even knowing whether the overall list succeeded.

I propose that in 'package list mode' (see 'go help test' for 'local directory mode' vs 'package list mode'), the overall 'go test' could end by printing FAIL when there are failures (but stay silent for all "ok"/"skip").

@markvincentcaro

This comment has been minimized.

Copy link
Author

markvincentcaro commented Mar 7, 2019

@rsc
Now this I can get behind. It would also be very helpful to put all failed tests, the specific line number, and t.Error messages at the bottom so we don't have to hunt for them at the top.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.