Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: Provide ability to inspect output of Example tests #41980

Open
komuw opened this issue Oct 14, 2020 · 5 comments
Open

proposal: testing: Provide ability to inspect output of Example tests #41980

komuw opened this issue Oct 14, 2020 · 5 comments
Labels
Projects
Milestone

Comments

@komuw
Copy link
Contributor

@komuw komuw commented Oct 14, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.1 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/home/Library/Caches/go-build"
GOENV="/Users/home/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/home/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/home/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/home/mystuff/kama/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/gc/sfs6hvtd1r392kn91n3jp17m0000gn/T/go-build416873732=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Created test example (https://golang.org/pkg/testing/#hdr-Examples, https://blog.golang.org/examples).
A reproducer of the issue is at https://play.golang.org/p/969hijP9sMx

What did you expect to see?

That example test is failing because there is a mismatch between what is been output to stdOut and what the test runner expects the output to be.
Unfortunately for test examples(unlike normal tests), I'm not able to programmatically get the test output so that I can examine(via diffing or otherwise) it and compare it to the expected result.

What did you see instead?

I do not have a way to access(programmatically) the output of the test.

@gopherbot gopherbot added this to the Proposal milestone Oct 14, 2020
@gopherbot gopherbot added the Proposal label Oct 14, 2020
@komuw
Copy link
Contributor Author

@komuw komuw commented Oct 14, 2020

The only way I was able to achieve what I want is via the following bad hack;

diff --git a/src/testing/run_example.go b/src/testing/run_example.go
index 10bde49e5b..b1c86f54d2 100644
--- a/src/testing/run_example.go
+++ b/src/testing/run_example.go
@@ -16,6 +16,8 @@ import (
        "os"
        "strings"
        "time"
+
+       "github.com/google/go-cmp/cmp"
 )
 func runExample(eg InternalExample) (ok bool) {
@@ -54,6 +56,12 @@ func runExample(eg InternalExample) (ok bool) {
                os.Stdout = stdout
                out := <-outC
+               got := strings.TrimSpace(out)
+               want := strings.TrimSpace(eg.Output)
+               diff := cmp.Diff(want, got)
+               fmt.Println("diff:")
+               fmt.Println(diff)
+
                err := recover()
                ok = eg.processRunResult(out, timeSpent, err)
        }()
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals Oct 14, 2020
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Oct 15, 2020

I do not have a way to access(programmatically) the output of the test.

Have you considered the -json flag of go test? For example:

{"Time":"2020-10-14T20:22:16.736222-04:00","Action":"output","Package":"m.test","Test":"ExampleHello","Output":"Hello \n"}
...
{"Time":"2020-10-14T20:22:16.736237-04:00","Action":"output","Package":"m.test","Test":"ExampleHello","Output":"Hello\n"}
@komuw
Copy link
Contributor Author

@komuw komuw commented Oct 15, 2020

Have you considered the -json flag of go test?

I guess that would work somehow. But I would have to run go test using another program that parses that output and compares the got and wanted parts of the output.

komuw added a commit to komuw/kama that referenced this issue Oct 15, 2020
What:
- Add example test. 
That test example is kind of blocked on;
1. golang/go#41980       
2. golang/go#5128 (comment)
@bcmills
Copy link
Member

@bcmills bcmills commented Oct 15, 2020

As another alternative, you could use os/exec to run os.Args[0] as a subprocess, to either run the generated ExampleHello test or invoke the original ExampleHello function directly. (See https://play.golang.org/p/T4fDo8D5M0H, but note that it doesn't work in the Playground due to #41339.)

You could either extract the want: output by parsing the output of the subprocess, or use go/doc.Examples to extract it programmatically.

@bcmills
Copy link
Member

@bcmills bcmills commented Oct 15, 2020

Come to think of it, it would be pretty straightforward to encapsulate the approach from the above comment as an external library. The API could look something like:

// RegisterExamples registers the given example functions so that they can be run by test functions.
//
// If this process is a subprocess started by ExampleOutput, RegisterExample
// runs the requested example function and then exits with status 0.
func RegisterExamples(m *testing.M, examples ...func())

// ExampleOutput returns the output of the given example function,
// obtained by executing the test executable as a subprocess.
//
// The function must have been passed to RegisterExamples in TestMain.
//
// If the current platform cannot execute subprocesses, RunExample skips t.
func RunExample(t testing.TB, example func()) (stderr, stdout string, err error)

// ExampleOutput returns the output of example, which must be an Example function
// recognized by 'go test'.
func ExampleOutput(t testing.TB, example func()) (want string)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.