I think this is pretty straightforward to fix for the typical case of a benchmark that produces no output.
If we see a --- PASS: TestCountToN line at the top level, I think we can assume that the output that follows does not belong with that test, although there are some old test outputs in cmd/internal/test2json/testdata (dating to Go 1.10) that violate that.
I suppose we need to figure out how and why those testdata outputs are structured that way.
Test2json is parsing the output stream from the test, which includes
package testing's own framing lines intermingled with other output,
in particular any output printed via fmt.Printf, println, and so on.
We have had recurring problems with unexpected partial output lines
causing a framing line to be missed.
A recent talk at GopherCon gave an example of an integration test
involving Docker that happened to print \r-terminated lines instead
of \n-terminated lines in some configurations, which in turn broke
test2json badly. (https://www.gophercon.com/agenda/session/944259)
There are also a variety of open reported issues with similar problems,
which this CL also addresses. The general approach is to add a new
testing flag -test.v=json that means to print additional output to help
test2json. And then test2json takes advantage of that output.
Among the fixes:
- Identify testing framing more reliably, using ^V
(golang#23036, golang#26325, golang#43683, GopherCon talk)
- Test that output with \r\n endings is handled correctly
- Use === RUN in fuzz tests (golang#52636, golang#48132)
- Add === RUN lines to note benchmark starts (golang#27764, golang#49505)
- Print subtest --- PASS/FAIL lines as they happen (golang#29811)
- Add === NAME lines to emit more test change events,
such as when a subtest stops and the parent continues running.
- Fix event shown in overall test failure (golang#27568)
- Avoid interleaving of writes to os.Stdout and os.Stderr (golang#33419)
Auto-Submit: Russ Cox <email@example.com>
Run-TryBot: Russ Cox <firstname.lastname@example.org>
Reviewed-by: Bryan Mills <email@example.com>
TryBot-Result: Gopher Robot <firstname.lastname@example.org>
What version of Go are you using (
Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (
What did you do?
We have three files:
I executed the following:
go test -json -shuffle off -bench '.+' -count 1 ./pkg/example
What did you expect to see?
A series of events in JSON format whose
Testproperty shows the name of the test/benchmark/example being executed. The benchmark should emit events bearing the field
What did you see instead?
The events corresponding to the benchmark are marked with the name of the preceding test (here:
pkg/example/x_test.gocauses the benchmark events to be annotated as
ExampleCountToNand leaving only the benchmark causes the
"Test"field to disappear from the output entirely:
It appears that the
Testfield of the
eventstruct is not populated at all for benchmarks, and the printer uses its last set value if one is available.
The text was updated successfully, but these errors were encountered: