Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: show diffs for incorrect output from Example tests #41980

Open
komuw opened this issue Oct 14, 2020 · 14 comments
Open

testing: show diffs for incorrect output from Example tests #41980

komuw opened this issue Oct 14, 2020 · 14 comments

Comments

@komuw
Copy link
Contributor

komuw commented Oct 14, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.1 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/home/Library/Caches/go-build"
GOENV="/Users/home/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/home/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/home/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/home/mystuff/kama/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gc/sfs6hvtd1r392kn91n3jp17m0000gn/T/go-build416873732=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Created test example (https://golang.org/pkg/testing/#hdr-Examples, https://blog.golang.org/examples).
A reproducer of the issue is at https://play.golang.org/p/969hijP9sMx

What did you expect to see?

That example test is failing because there is a mismatch between what is been output to stdOut and what the test runner expects the output to be.
Unfortunately for test examples(unlike normal tests), I'm not able to programmatically get the test output so that I can examine(via diffing or otherwise) it and compare it to the expected result.

What did you see instead?

I do not have a way to access(programmatically) the output of the test.

@gopherbot gopherbot added this to the Proposal milestone Oct 14, 2020
@komuw
Copy link
Contributor Author

komuw commented Oct 14, 2020

The only way I was able to achieve what I want is via the following bad hack;

diff --git a/src/testing/run_example.go b/src/testing/run_example.go
index 10bde49e5b..b1c86f54d2 100644
--- a/src/testing/run_example.go
+++ b/src/testing/run_example.go
@@ -16,6 +16,8 @@ import (
        "os"
        "strings"
        "time"
+
+       "github.com/google/go-cmp/cmp"
 )
 func runExample(eg InternalExample) (ok bool) {
@@ -54,6 +56,12 @@ func runExample(eg InternalExample) (ok bool) {
                os.Stdout = stdout
                out := <-outC
+               got := strings.TrimSpace(out)
+               want := strings.TrimSpace(eg.Output)
+               diff := cmp.Diff(want, got)
+               fmt.Println("diff:")
+               fmt.Println(diff)
+
                err := recover()
                ok = eg.processRunResult(out, timeSpent, err)
        }()

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Oct 14, 2020
@dmitshur
Copy link
Contributor

dmitshur commented Oct 15, 2020

I do not have a way to access(programmatically) the output of the test.

Have you considered the -json flag of go test? For example:

{"Time":"2020-10-14T20:22:16.736222-04:00","Action":"output","Package":"m.test","Test":"ExampleHello","Output":"Hello \n"}
...
{"Time":"2020-10-14T20:22:16.736237-04:00","Action":"output","Package":"m.test","Test":"ExampleHello","Output":"Hello\n"}

@komuw
Copy link
Contributor Author

komuw commented Oct 15, 2020

Have you considered the -json flag of go test?

I guess that would work somehow. But I would have to run go test using another program that parses that output and compares the got and wanted parts of the output.

komuw added a commit to komuw/kama that referenced this issue Oct 15, 2020
What:
- Add example test. 
That test example is kind of blocked on;
1. golang/go#41980       
2. golang/go#5128 (comment)
@bcmills
Copy link
Member

bcmills commented Oct 15, 2020

As another alternative, you could use os/exec to run os.Args[0] as a subprocess, to either run the generated ExampleHello test or invoke the original ExampleHello function directly. (See https://play.golang.org/p/T4fDo8D5M0H, but note that it doesn't work in the Playground due to #41339.)

You could either extract the want: output by parsing the output of the subprocess, or use go/doc.Examples to extract it programmatically.

@bcmills
Copy link
Member

bcmills commented Oct 15, 2020

Come to think of it, it would be pretty straightforward to encapsulate the approach from the above comment as an external library. The API could look something like:

// RegisterExamples registers the given example functions so that they can be run by test functions.
//
// If this process is a subprocess started by ExampleOutput, RegisterExample
// runs the requested example function and then exits with status 0.
func RegisterExamples(m *testing.M, examples ...func())

// ExampleOutput returns the output of the given example function,
// obtained by executing the test executable as a subprocess.
//
// The function must have been passed to RegisterExamples in TestMain.
//
// If the current platform cannot execute subprocesses, RunExample skips t.
func RunExample(t testing.TB, example func()) (stderr, stdout string, err error)

// ExampleOutput returns the output of example, which must be an Example function
// recognized by 'go test'.
func ExampleOutput(t testing.TB, example func()) (want string)

@rsc
Copy link
Contributor

rsc commented Oct 21, 2020

There is a trap here, namely adding infinite new API to package testing as individual needs come up.
We need testing to do a limited amount of things. It's already getting hard to remember everything.

Looking at https://play.golang.org/p/969hijP9sMx, the issue I see is not that testing needs new API but instead that the example test output needs to do a better job highlighting the differences, such as by showing a diff. Maybe we should do that instead?

@rsc rsc moved this from Incoming to Active in Proposals (old) Oct 21, 2020
@komuw
Copy link
Contributor Author

komuw commented Oct 22, 2020

Looking at https://play.golang.org/p/969hijP9sMx, the issue I see is not that testing needs new API but instead that the example test output needs to do a better job highlighting the differences, such as by showing a diff. Maybe we should do that instead?

That would be excellent.

@rsc rsc changed the title proposal: testing: Provide ability to inspect output of Example tests proposal: testing: show diffs for incorrect output from Example tests Oct 28, 2020
@rsc
Copy link
Contributor

rsc commented Oct 28, 2020

Retitled for the suggestion I made last week, namely show diffs.
Does anyone object to that?

@mvdan
Copy link
Member

mvdan commented Oct 29, 2020

What diff package would we use for this? gofmt -d just relies on the system's diff program, but I don't think that's a good idea for testing.

@josharian has been working on https://github.com/pkg/diff, so perhaps that could be vendored as an internal package.

@rsc
Copy link
Contributor

rsc commented Nov 4, 2020

We can find a reasonable diff package. I have a few sitting around too.

@rsc
Copy link
Contributor

rsc commented Nov 4, 2020

Based on the discussion, this seems like a likely accept.

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Nov 4, 2020
@rsc
Copy link
Contributor

rsc commented Nov 11, 2020

No change in consensus, so accepted.

@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Nov 11, 2020
@rsc rsc changed the title proposal: testing: show diffs for incorrect output from Example tests testing: show diffs for incorrect output from Example tests Nov 11, 2020
@rsc rsc modified the milestones: Proposal, Backlog Nov 11, 2020
@odeke-em odeke-em self-assigned this Mar 12, 2021
@gopherbot
Copy link

gopherbot commented Mar 14, 2021

Change https://golang.org/cl/301589 mentions this issue: testing: add diffing to clearly discern Example test failure output

@mvdan
Copy link
Member

mvdan commented Feb 18, 2022

I've just filed #51254, which is a pretty similar proposal, FYI :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Accepted
Development

No branches or pull requests

7 participants