Repository navigation
testing: streaming output loses parallel subtest associations #38458
Description
Activity
- changed the title
[-]testing: streaming output loses subtest associations[/-][+]testing: streaming output loses parallel subtest associations[/+]on Apr 15, 2020 Isn't the
TestTestdata/MIT-0.t1:prefix in the parallel output the full subtest path?TestTestdata/MIT-0.t1: license_test.go:138: testdata/MIT-0.t1:1,3: ↑ subtest name ↑ source location ↑ start of user-generated message
I didn't notice the prefix. Having that prefix on every line is pretty noisy. It's turned the output into something that requires a machine to read. I think we should change this output again. I know we just did in Go 1.14, but the current output is simply not human-friendly. The prefix is stuttering, and "file:line:" should always be at the start of a line.
$ cat x_test.go package main import "testing" func Test(t *testing.T) { for i := 0; i < 10; i++ { t.Log(i, "->\n", i) } } $ go test -v x_test.go === RUN Test Test: x_test.go:7: 0 -> 0 Test: x_test.go:7: 1 -> 1 Test: x_test.go:7: 2 -> 2 Test: x_test.go:7: 3 -> 3 Test: x_test.go:7: 4 -> 4 Test: x_test.go:7: 5 -> 5 Test: x_test.go:7: 6 -> 6 Test: x_test.go:7: 7 -> 7 Test: x_test.go:7: 8 -> 8 Test: x_test.go:7: 9 -> 9 --- PASS: Test (0.00s) PASS ok command-line-arguments 0.075s $ go1.12 test -v x_test.go === RUN Test --- PASS: Test (0.00s) x_test.go:7: 0 -> 0 x_test.go:7: 1 -> 1 x_test.go:7: 2 -> 2 x_test.go:7: 3 -> 3 x_test.go:7: 4 -> 4 x_test.go:7: 5 -> 5 x_test.go:7: 6 -> 6 x_test.go:7: 7 -> 7 x_test.go:7: 8 -> 8 x_test.go:7: 9 -> 9 PASS ok command-line-arguments (cached) $Reacted by Bryan C. Mills and Daniel NephinLet's replace the prefix with tracking what test name has been printed in an === line most recently, and print a new === line if needed to change the "currently printing" test.
This will change the work done for #24929
CC @jadekler @randall77
One consequence of #24929 (streaming t.Log output) is that
test2jsonwasn't updated with any understanding of streaming output and how it would affect parallel subtests. FWIW, PR #38189 (for issue #38050) attempts to correct that oversight. This comment is a friendly reminder to include thetest2jsonutility in whatever changes result from this issue.Reacted by Dmitri ShuralyovChange https://golang.org/cl/229085 mentions this issue:
testing: print === lines when subtests change output in parallel modeI spent time looking into this today, and am working on a cl that implements @rsc's suggestion: https://go-review.googlesource.com/c/go/+/229085. Russ, your solution seems reasonable. Here's what that looks like:
Before
$ go test . -v === RUN TestFoo === RUN TestFoo/SubTest-2 TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-0) TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-1) === RUN TestFoo/SubTest-0 TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-0) === RUN TestFoo/SubTest-1 TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-0) TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-2) TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-1) TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-3) TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-2) TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-1) TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-4) TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-2) TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-3) TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-3) TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-4) TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-4) --- PASS: TestFoo (0.00s) --- PASS: TestFoo/SubTest-2 (0.00s) --- PASS: TestFoo/SubTest-1 (0.00s) --- PASS: TestFoo/SubTest-0 (0.00s)After
=== RUN TestName === RUN TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-0) sub_test.go:571: This is a t.Log from sub-2 (message-1) === RUN TestName/SubTest-0 === RUN TestName/SubTest-1 === TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-0) === TestName/SubTest-1 sub_test.go:571: This is a t.Log from sub-1 (message-0) === TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-2) === TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-1) sub_test.go:571: This is a t.Log from sub-0 (message-2) === TestName/SubTest-1 sub_test.go:571: This is a t.Log from sub-1 (message-1) === TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-3) === TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-3) === TestName/SubTest-1 sub_test.go:571: This is a t.Log from sub-1 (message-2) sub_test.go:571: This is a t.Log from sub-1 (message-3) sub_test.go:571: This is a t.Log from sub-1 (message-4) === TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-4) === TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-4) --- PASS: TestName (0.00s) --- PASS: TestName/SubTest-1 (0.00s) --- PASS: TestName/SubTest-2 (0.00s)Does this look right? As you can see, it's quite a bit more chatty when there's a high degree of interleaving, which could be a bummer. On the other hand, it does separate subtests better.
If so, happy to continue on that CL and figure out a real way to implement this, instead of my hacky way. 🙂 (I also figured out a better way to capture and test subtest output, so minimally I'd like to get that into the testsuite)
The indented === lines look wrong. They have no verb and they are indented.
Also, how is it even possible to be running multiple tests without a === PAUSE line?OK, I looked at the CL test. Starting a subtest inside a kicked-off goroutine is non-standard to say the least. The important part is that the output should not require any changes to test2json. Test2json expects === lines to be unindented and contain a verb. The obvious verb here is "CONT" for continue, so that no new verb is needed either.
Instead of introducing the output redirect in package testing, I'd rather see a test script in cmd/go, testing both go test -v and go test -json. Then the diff should be very small.
That is, I think the output should be:
=== RUN TestName === RUN TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-0) sub_test.go:571: This is a t.Log from sub-2 (message-1) === RUN TestName/SubTest-0 === RUN TestName/SubTest-1 === CONT TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-0) === CONT TestName/SubTest-1 sub_test.go:571: This is a t.Log from sub-1 (message-0) === CONT TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-2) === CONT TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-1) sub_test.go:571: This is a t.Log from sub-0 (message-2) === CONT TestName/SubTest-1 sub_test.go:571: This is a t.Log from sub-1 (message-1) === CONT TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-3) === CONT TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-3) === CONT TestName/SubTest-1 sub_test.go:571: This is a t.Log from sub-1 (message-2) sub_test.go:571: This is a t.Log from sub-1 (message-3) sub_test.go:571: This is a t.Log from sub-1 (message-4) === CONT TestName/SubTest-2 sub_test.go:571: This is a t.Log from sub-2 (message-4) === CONT TestName/SubTest-0 sub_test.go:571: This is a t.Log from sub-0 (message-4) --- PASS: TestName (0.00s) --- PASS: TestName/SubTest-1 (0.00s) --- PASS: TestName/SubTest-2 (0.00s)Starting a subtest inside a kicked-off goroutine is non-standard to say the least.
t.Parallel was blocking for some reason, so after some investigation I threw my hands up and went with the easy out to get at what I wanted. And, it was a prototype CL, so wasn't too concerned. But good to know that's unexpected for future tests!
Instead of introducing the output redirect in package testing, I'd rather see a test script in cmd/go, testing both go test -v and go test -json. Then the diff should be very small.
SG, thanks for the direction! That helps me orient.
That is, I think the output should be:
SG, thanks for the output. I'll work on updating the CL. 👍
I've updated https://go-review.googlesource.com/c/go/+/229085:
- I've put the test in cmd/go/testdata/script. (still need a benchmark test, and a -json test)
- I've updated the implementation to work correctly when FAIL and PASS occurs.
- I've removed some of the old prototype stuff.
I'm not in love with the solution, though. It seems not-great to lock, record, and unlock everywhere we print. That seems like something that's easy to forget to do in the future. I explored adding a writer that keeps track of the test name on all outgoing writes, but quickly realized it would have to interpolate each outgoing line to figure out test name (no way to pass test name through io.Writer.Write AFAICT), which doesn't seem great.
Would love thoughts if there's a better approach. (also on anything else, but particularly that)
I explored adding a writer that keeps track of the test name on all outgoing writes, but quickly realized it would have to interpolate each outgoing line to figure out test name (no way to pass test name through io.Writer.Write AFAICT), which doesn't seem great.
Don't do it as a generic
Writer. Could you implement it as a method that takes a prefix, name, suffix format, and suffix args?Reacted by Jean Barkhuysen25 remaining items
Change https://golang.org/cl/242057 mentions this issue:
[release-branch.go1.14] testing: reformat test chatty outputChange https://golang.org/cl/242059 mentions this issue:
[release-branch.go1.14] cmd/go: fix parallel chatty tests on solaris-amd64 builder- added a commit that references this issue
on Jul 21, 2020 Change https://golang.org/cl/249026 mentions this issue:
testing: flush test summaries to stdout atomically when streaming output- added a commit that references this issue
on Sep 2, 2020 Change https://golang.org/cl/252637 mentions this issue:
[release-branch.go1.15] testing: flush test summaries to stdout atomically when streaming outputChange https://golang.org/cl/252638 mentions this issue:
[release-branch.go1.14] testing: flush test summaries to stdout atomically when streaming output- added a commit that references this issue
on Oct 22, 2020 - locked and limited conversation to collaborators
on Sep 2, 2021
Normal
go testprints output like this for parallel subtests:My memory is that we made a change to try to print output earlier for
-v, but this is completely breaking the output for parallel subtests. There is no obvious way to tell which test printed:In this case, the subtest name happens to be echoed in the message, but that's because of a print in the test; the testing framework has failed at its job and omitted the information.
In contrast, here is Go 1.12:
And here is current go with the
t.Parallelcall commented out:Rolling back the change for parallel subtests would be unfortunate, since it would make the output placement dependent on use of
t.Parallel. (It's already unfortunate that the output placement is dependent on-vbut that ship has sailed.)Perhaps the best fix would be to print a new
=== CONTline each time the test generating output changes (before the output).