You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
proposal: testing: add -json flag for json results #2981
Many build servers also like the JUnit XML format. Might be worth it to do both. We've
had good success with running go test -v and translating to the JUnit format (it even
has a place to put stdout and stderr).
Reacted by anatoly techtonik, Phil Kates, Richard Strand and Emanuel Evans
Let's do this. Plan of attack:
Structs for output:
type Result struct {
Tests []TestResult
Benchmarks []BenchmarkResult
About struct {
Version string
OS string
Arch string
Hostname string
StartTime time.Time
EndTime time.Time
}
}
type BenchmarkResult +=
Package string
Name string
type TestResult struct {
Package string
Name string
Pass bool
Output string
Duration time.Duration
}
Add flag -test.package=math/rand (import path of package), for use in setting Package in
TestResult and BenchmarkResult.
Add -test.json (bool flag) to generate json instead of standard output.
Generated json should be indented.
Any interest in generalising this slightly to allow a JUnit format to be contributed
later?
E.g.
-test.output=std (default)
-test.output=json
-test.output=junitxml
I would like to make sure we have the information JUnit needs, but I'm
not interested in generating the XML directly. I'd rather that
separate tools took care of externally-defined formats.
Russ
Russ, what is the purpose of -test.package? Shouldn't 'go test' pass the import path of
the package under test to package testing? Would {Benchmark,Test}Result.Package be blank
when -test.package isn't set?
The idea is that 'go test' would specify -test.package on the command line when running
the test.
However, it could also compile it in, by adding Package string fields to
testing.InternalTest and testing.InternalBenchmark. That's probably better than the
command-line option.
I've got the beginning sketches of an implementation at
https://golang.org/cl/7061048/. I'm wondering, though, what the interaction of
-test.json with cmd/go/test should be.
- If we print out summary info (e.g., ok archive/tar 0.006s), then the output of 'go
test' is not valid json -- you'd have to pre-process it a bit.
- Ditto for multiple packages, if we just print out the results sequentially for each
package.
But, the flag is called -test.json, so 'go test' doesn't directly handle it, right?
Similarly, what should happen if -test.v and -test.json are both set? I'd suggest that
-test.json disable chattiness because the output is part of the json result anyway.
In json mode the testing framework should not print about starting and
stopping each test. It cannot clear test.v, though, because tests
might use testing.Verbose() (which looks at test.v) to decide how much
log output to generate, and we do want to generate that output (and
save it into the JSON).
The go command will need to know about the json flag regardless, in
order to turn -json into -test.json. When in that mode, it should
synthesize its own Result built containing all the TestResults and
BenchmarkResults from the individual packages. The ability to do this
is the reason that Package appears in each TestResult/BenchmarkResult
instead of in the top-level Result.
Russ
I spent a while looking into this. I think it's too complex to get into Go 1.1. The hard
part is making sure that every possible case ends up printing json and not text intended
for humans.
First there is the problem that all test output needs to be caught, even output
generated with fmt.Printf or the builtin println or panic traces, and even if the test
binary exits unexpectedly. The only truly feasible way to do this is to make the go
command interpret the test binary output and generate the JSON from that interpretation.
I started to try to make that a little easier by adding a -test.outputprefix flag to
specify a prefix string that the test binary will put at the beginning of each line that
it generates, so that the structure of the test output can be seen more readily. That's
in CL 7678044.
But then the next problem is making the go command parse that output and generate JSON.
And the go command must also catch all its own output and convert it to JSON
equivalents, not just for running tests but also for build failures and the like. That's
a bigger task, and at some point it starts to seem like maybe we should make the go
command just have a little bit of framing and have a separate program whose only job is
to invoke the go command (or, perhaps, also a test binary directly) and parse the output
and turn it into JSON.
This is too big a change for what time we have remaining for Go 1.1. Perhaps we can
figure out what the story is by Go 1.2. For now, an external tool that parses the go
command output (and therefore also the test binary output, if using go test -v) seems
like the best choice. And then once that exists we can think about small framing changes
to make the parsing less confusable.
@rsc I think the tool can be maintained by the community / be an golang.org/x tool as long as issues like #19280 or any others which prevent the test reader from correctly determining which test writes which output are addressed. What do you think?
The subject line on this issue is to add a -json flag. If there's a -json flag, it needs to use a tool from the standard repo - we want Go to be self-contained.
I've tried to apply this feature in GoLand and this is the list of issues I faced:
-json parameter doesn't work with -c argument
-json parameter doesn't work with go bench. The only json output for benchmarks is package-passed event (btw, it's incorrectly "failed", looks like a bug), no information about starting, ending benchmarks or about text in stdout in json format.
PASS
ok github.com/rcrowley/go-metrics 40.797s
{"Time":"2017-11-11T12:55:29.076331942+03:00","Action":"fail","Package":"github.com/rcrowley/go-metrics","Elapsed":40.797}
…
Some events have become a part of output of others, like this one:
/home/zolotov/dev/go/bin/go test -v -json ./...
{"Time":"2017-11-11T13:05:28.267338195+03:00","Action":"run","Package":"github.com/rcrowley/go-metrics","Test":"TestCounterClear"}
{"Time":"2017-11-11T13:05:28.26745939+03:00","Action":"output","Package":"github.com/rcrowley/go-metrics","Test":"TestCounterClear","Output":"=== RUN TestCounterClear\n"}
{"Time":"2017-11-11T13:05:28.267468575+03:00","Action":"output","Package":"github.com/rcrowley/go-metrics","Test":"TestCounterClear","Output":"{"Time":"2017-11-11T13:05:28.267472348+03:00","Action":"pass","Package":"github.com/rcrowley/go-metrics","Test":"TestCounterClear","Elapsed":0}
{"Time":"2017-11-11T13:05:28.267476892+03:00","Action":"run","Package":"github.com/rcrowley/go-metrics","Test":"TestCounterDec1"}
Activity
alberts commentedon Mar 12, 2012
Comment 1:
rsc commentedon Sep 12, 2012
Comment 2:
Labels changed: added go1.1.
rsc commentedon Dec 10, 2012
Comment 3:
Let's do this. Plan of attack: Structs for output: type Result struct { Tests []TestResult Benchmarks []BenchmarkResult About struct { Version string OS string Arch string Hostname string StartTime time.Time EndTime time.Time } } type BenchmarkResult += Package string Name string type TestResult struct { Package string Name string Pass bool Output string Duration time.Duration } Add flag -test.package=math/rand (import path of package), for use in setting Package in TestResult and BenchmarkResult. Add -test.json (bool flag) to generate json instead of standard output. Generated json should be indented.rsc commentedon Dec 10, 2012
Comment 4:
Labels changed: added size-l.
rsc commentedon Dec 10, 2012
Comment 5:
Labels changed: added suggested.
alberts commentedon Dec 10, 2012
Comment 6:
rsc commentedon Dec 11, 2012
Comment 7:
alberts commentedon Dec 12, 2012
Comment 8:
tebeka commentedon Dec 12, 2012
Comment 9:
cespare commentedon Jan 6, 2013
Comment 10:
Russ, what is the purpose of -test.package? Shouldn't 'go test' pass the import path of the package under test to package testing? Would {Benchmark,Test}Result.Package be blank when -test.package isn't set?rsc commentedon Jan 7, 2013
Comment 11:
cespare commentedon Jan 7, 2013
Comment 12:
rsc commentedon Jan 9, 2013
Comment 13:
rsc commentedon Mar 11, 2013
Comment 14:
Labels changed: added go1.2, removed go1.1.
rsc commentedon Mar 12, 2013
Comment 15:
123 remaining items
dlsniper commentedon Oct 9, 2017
@rsc I think the tool can be maintained by the community / be an golang.org/x tool as long as issues like #19280 or any others which prevent the test reader from correctly determining which test writes which output are addressed. What do you think?
rsc commentedon Oct 9, 2017
The subject line on this issue is to add a -json flag. If there's a -json flag, it needs to use a tool from the standard repo - we want Go to be self-contained.
gopherbot commentedon Nov 10, 2017
Change https://golang.org/cl/76873 mentions this issue:
cmd/go: add go test -json flaggopherbot commentedon Nov 10, 2017
Change https://golang.org/cl/76872 mentions this issue:
cmd/test2json: go tool test2json converts test output to JSONcmd/test2json: go tool test2json converts test output to JSON
zolotov commentedon Nov 14, 2017
I've tried to apply this feature in GoLand and this is the list of issues I faced:
-jsonparameter doesn't work with-cargument-jsonparameter doesn't work withgo bench. The only json output for benchmarks is package-passed event (btw, it's incorrectly "failed", looks like a bug), no information about starting, ending benchmarks or about text in stdout in json format.The third line is barely parseable.
ianlancetaylor commentedon Nov 14, 2017
@zolotov Would you be willing to open new issues for the problems you have found? Thanks.
zolotov commentedon Nov 15, 2017
@iangudger I can but I believe these problems show that this issue is not solved, what's the point of creating one more "support json output" issue?
bradfitz commentedon Nov 15, 2017
@zolotov, you got the wrong Ian.
We don't reuse bugs. The bug is closed and is not tracked on our dashboards, so comments here are likely to go into the void and never be seen.
ianlancetaylor commentedon Nov 15, 2017
@zolotov The general problem of JSON output is, we believe, solved. You are talking about specific problems with the solution.
bradfitz commentedon Nov 15, 2017
I filed #22734
zolotov commentedon Nov 15, 2017
@bradfitz got it. thanks