Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: streaming output loses parallel subtest associations #38458

Closed
rsc opened this issue Apr 15, 2020 · 32 comments
Closed

testing: streaming output loses parallel subtest associations #38458

rsc opened this issue Apr 15, 2020 · 32 comments

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Apr 15, 2020

Normal go test prints output like this for parallel subtests:

$ go test -run=Testdata/MIT
--- FAIL: TestTestdata (0.51s)
    --- FAIL: TestTestdata/MIT-0.t1 (0.00s)
        license_test.go:138: testdata/MIT-0.t1:1,3: diff -want +have:
            - 93.5%
            + 97.4%
            - MIT-0 100.0% 74,$
            + MIT-0 100.0% 26,$
    --- FAIL: TestTestdata/MIT-0.t2 (0.00s)
        license_test.go:138: testdata/MIT-0.t2:2,4: diff -want +have:
            - 100.0%
            + 0.0%
            - MIT-0 100.0% 0,$
    --- FAIL: TestTestdata/MIT.t3 (0.00s)
        license_test.go:138: testdata/MIT.t3:2,4: diff -want +have:
            - 100.0%
            + 0.0%
            - MIT 95.8% 0,$
    --- FAIL: TestTestdata/MIT-Header.t1 (0.00s)
        license_test.go:138: testdata/MIT-Header.t1:1,3: diff -want +have:
            - 86.9%
            + 0.0%
            - MIT-Header 100.0% 83,$
FAIL

My memory is that we made a change to try to print output earlier for -v, but this is completely breaking the output for parallel subtests. There is no obvious way to tell which test printed:

$ go test -v -run=Testdata/MIT
=== RUN   TestTestdata
=== RUN   TestTestdata/MIT-0.t1
=== PAUSE TestTestdata/MIT-0.t1
=== RUN   TestTestdata/MIT-0.t2
=== PAUSE TestTestdata/MIT-0.t2
=== RUN   TestTestdata/MIT-Header.t1
=== PAUSE TestTestdata/MIT-Header.t1
=== RUN   TestTestdata/MIT.t1
=== PAUSE TestTestdata/MIT.t1
=== RUN   TestTestdata/MIT.t2
=== PAUSE TestTestdata/MIT.t2
=== RUN   TestTestdata/MIT.t3
=== PAUSE TestTestdata/MIT.t3
=== CONT  TestTestdata/MIT-0.t1
=== CONT  TestTestdata/MIT.t1
=== CONT  TestTestdata/MIT-0.t2
=== CONT  TestTestdata/MIT.t3
=== CONT  TestTestdata/MIT.t2
=== CONT  TestTestdata/MIT-Header.t1
    TestTestdata/MIT-0.t1: license_test.go:138: testdata/MIT-0.t1:1,3: diff -want +have:
        - 93.5%
        + 97.4%
        - MIT-0 100.0% 74,$
        + MIT-0 100.0% 26,$
    TestTestdata/MIT-0.t2: license_test.go:138: testdata/MIT-0.t2:2,4: diff -want +have:
        - 100.0%
        + 0.0%
        - MIT-0 100.0% 0,$
    TestTestdata/MIT.t3: license_test.go:138: testdata/MIT.t3:2,4: diff -want +have:
        - 100.0%
        + 0.0%
        - MIT 95.8% 0,$
    TestTestdata/MIT-Header.t1: license_test.go:138: testdata/MIT-Header.t1:1,3: diff -want +have:
        - 86.9%
        + 0.0%
        - MIT-Header 100.0% 83,$
--- FAIL: TestTestdata (0.52s)
    --- FAIL: TestTestdata/MIT-0.t1 (0.00s)
    --- FAIL: TestTestdata/MIT-0.t2 (0.00s)
    --- PASS: TestTestdata/MIT.t1 (0.00s)
    --- PASS: TestTestdata/MIT.t2 (0.00s)
    --- FAIL: TestTestdata/MIT.t3 (0.00s)
    --- FAIL: TestTestdata/MIT-Header.t1 (0.00s)
FAIL

In this case, the subtest name happens to be echoed in the message, but that's because of a print in the test; the testing framework has failed at its job and omitted the information.

In contrast, here is Go 1.12:

$ go1.12 test -run=Testdata/MIT -v
=== RUN   TestTestdata
=== RUN   TestTestdata/MIT-0.t1
=== PAUSE TestTestdata/MIT-0.t1
=== RUN   TestTestdata/MIT-0.t2
=== PAUSE TestTestdata/MIT-0.t2
=== RUN   TestTestdata/MIT-Header.t1
=== PAUSE TestTestdata/MIT-Header.t1
=== RUN   TestTestdata/MIT.t1
=== PAUSE TestTestdata/MIT.t1
=== RUN   TestTestdata/MIT.t2
=== PAUSE TestTestdata/MIT.t2
=== RUN   TestTestdata/MIT.t3
=== PAUSE TestTestdata/MIT.t3
=== CONT  TestTestdata/MIT-0.t1
=== CONT  TestTestdata/MIT.t1
=== CONT  TestTestdata/MIT.t2
=== CONT  TestTestdata/MIT-Header.t1
=== CONT  TestTestdata/MIT-0.t2
=== CONT  TestTestdata/MIT.t3
--- FAIL: TestTestdata (0.57s)
    --- PASS: TestTestdata/MIT.t1 (0.00s)
    --- FAIL: TestTestdata/MIT-0.t1 (0.00s)
        license_test.go:138: testdata/MIT-0.t1:1,3: diff -want +have:
            - 93.5%
            + 97.4%
            - MIT-0 100.0% 74,$
            + MIT-0 100.0% 26,$
    --- PASS: TestTestdata/MIT.t2 (0.00s)
    --- FAIL: TestTestdata/MIT-Header.t1 (0.00s)
        license_test.go:138: testdata/MIT-Header.t1:1,3: diff -want +have:
            - 86.9%
            + 0.0%
            - MIT-Header 100.0% 83,$
    --- FAIL: TestTestdata/MIT.t3 (0.00s)
        license_test.go:138: testdata/MIT.t3:2,4: diff -want +have:
            - 100.0%
            + 0.0%
            - MIT 95.8% 0,$
    --- FAIL: TestTestdata/MIT-0.t2 (0.00s)
        license_test.go:138: testdata/MIT-0.t2:2,4: diff -want +have:
            - 100.0%
            + 0.0%
            - MIT-0 100.0% 0,$
FAIL
exit status 1
FAIL	github.com/google/licensecheck	0.882s
$ 

And here is current go with the t.Parallel call commented out:

$ go test -run=Testdata/MIT -v
=== RUN   TestTestdata
=== RUN   TestTestdata/MIT-0.t1
    TestTestdata/MIT-0.t1: license_test.go:138: testdata/MIT-0.t1:1,3: diff -want +have:
        - 93.5%
        + 97.4%
        - MIT-0 100.0% 74,$
        + MIT-0 100.0% 26,$
=== RUN   TestTestdata/MIT-0.t2
    TestTestdata/MIT-0.t2: license_test.go:138: testdata/MIT-0.t2:2,4: diff -want +have:
        - 100.0%
        + 0.0%
        - MIT-0 100.0% 0,$
=== RUN   TestTestdata/MIT-Header.t1
    TestTestdata/MIT-Header.t1: license_test.go:138: testdata/MIT-Header.t1:1,3: diff -want +have:
        - 86.9%
        + 0.0%
        - MIT-Header 100.0% 83,$
=== RUN   TestTestdata/MIT.t1
=== RUN   TestTestdata/MIT.t2
=== RUN   TestTestdata/MIT.t3
    TestTestdata/MIT.t3: license_test.go:138: testdata/MIT.t3:2,4: diff -want +have:
        - 100.0%
        + 0.0%
        - MIT 95.8% 0,$
--- FAIL: TestTestdata (0.54s)
    --- FAIL: TestTestdata/MIT-0.t1 (0.00s)
    --- FAIL: TestTestdata/MIT-0.t2 (0.00s)
    --- FAIL: TestTestdata/MIT-Header.t1 (0.00s)
    --- PASS: TestTestdata/MIT.t1 (0.00s)
    --- PASS: TestTestdata/MIT.t2 (0.00s)
    --- FAIL: TestTestdata/MIT.t3 (0.00s)
FAIL
$ 

Rolling back the change for parallel subtests would be unfortunate, since it would make the output placement dependent on use of t.Parallel. (It's already unfortunate that the output placement is dependent on -v but that ship has sailed.)

Perhaps the best fix would be to print a new === CONT line each time the test generating output changes (before the output).

@rsc rsc added the NeedsFix label Apr 15, 2020
@rsc rsc added this to the Go1.15 milestone Apr 15, 2020
@rsc rsc changed the title testing: streaming output loses subtest associations testing: streaming output loses parallel subtest associations Apr 15, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Apr 15, 2020

Isn't the TestTestdata/MIT-0.t1: prefix in the parallel output the full subtest path?

	TestTestdata/MIT-0.t1: license_test.go:138: testdata/MIT-0.t1:1,3:
	↑ subtest name         ↑ source location    ↑ start of user-generated message
@rsc
Copy link
Contributor Author

@rsc rsc commented Apr 16, 2020

I didn't notice the prefix. Having that prefix on every line is pretty noisy. It's turned the output into something that requires a machine to read. I think we should change this output again. I know we just did in Go 1.14, but the current output is simply not human-friendly. The prefix is stuttering, and "file:line:" should always be at the start of a line.

$ cat x_test.go
package main

import "testing"

func Test(t *testing.T) {
	for i := 0; i < 10; i++ {
		t.Log(i, "->\n", i)
	}
}
$ go test -v x_test.go
=== RUN   Test
    Test: x_test.go:7: 0 ->
         0
    Test: x_test.go:7: 1 ->
         1
    Test: x_test.go:7: 2 ->
         2
    Test: x_test.go:7: 3 ->
         3
    Test: x_test.go:7: 4 ->
         4
    Test: x_test.go:7: 5 ->
         5
    Test: x_test.go:7: 6 ->
         6
    Test: x_test.go:7: 7 ->
         7
    Test: x_test.go:7: 8 ->
         8
    Test: x_test.go:7: 9 ->
         9
--- PASS: Test (0.00s)
PASS
ok  	command-line-arguments	0.075s
$ go1.12 test -v x_test.go
=== RUN   Test
--- PASS: Test (0.00s)
    x_test.go:7: 0 ->
         0
    x_test.go:7: 1 ->
         1
    x_test.go:7: 2 ->
         2
    x_test.go:7: 3 ->
         3
    x_test.go:7: 4 ->
         4
    x_test.go:7: 5 ->
         5
    x_test.go:7: 6 ->
         6
    x_test.go:7: 7 ->
         7
    x_test.go:7: 8 ->
         8
    x_test.go:7: 9 ->
         9
PASS
ok  	command-line-arguments	(cached)
$ 
@rsc
Copy link
Contributor Author

@rsc rsc commented Apr 16, 2020

Let's replace the prefix with tracking what test name has been printed in an === line most recently, and print a new === line if needed to change the "currently printing" test.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 17, 2020

This will change the work done for #24929

CC @jadekler @randall77

@mdwhatcott
Copy link

@mdwhatcott mdwhatcott commented Apr 18, 2020

One consequence of #24929 (streaming t.Log output) is that test2json wasn't updated with any understanding of streaming output and how it would affect parallel subtests. FWIW, PR #38189 (for issue #38050) attempts to correct that oversight. This comment is a friendly reminder to include the test2json utility in whatever changes result from this issue.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 21, 2020

Change https://golang.org/cl/229085 mentions this issue: testing: print === lines when subtests change output in parallel mode

@jadekler
Copy link
Contributor

@jadekler jadekler commented Apr 21, 2020

I spent time looking into this today, and am working on a cl that implements @rsc's suggestion: https://go-review.googlesource.com/c/go/+/229085. Russ, your solution seems reasonable. Here's what that looks like:

Before

$ go test . -v
=== RUN   TestFoo
=== RUN   TestFoo/SubTest-2
    TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-0)
    TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-1)
=== RUN   TestFoo/SubTest-0
    TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-0)
=== RUN   TestFoo/SubTest-1
    TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-0)
    TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-2)
    TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-1)
    TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-3)
    TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-2)
    TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-1)
    TestFoo/SubTest-2: foo_test.go:23: This is a t.Log from sub-2 (message-4)
    TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-2)
    TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-3)
    TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-3)
    TestFoo/SubTest-1: foo_test.go:23: This is a t.Log from sub-1 (message-4)
    TestFoo/SubTest-0: foo_test.go:23: This is a t.Log from sub-0 (message-4)
--- PASS: TestFoo (0.00s)
    --- PASS: TestFoo/SubTest-2 (0.00s)
    --- PASS: TestFoo/SubTest-1 (0.00s)
    --- PASS: TestFoo/SubTest-0 (0.00s)

After

=== RUN   TestName
=== RUN   TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-0)
        sub_test.go:571: This is a t.Log from sub-2 (message-1)
=== RUN   TestName/SubTest-0
=== RUN   TestName/SubTest-1
    === TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-0)
    === TestName/SubTest-1
        sub_test.go:571: This is a t.Log from sub-1 (message-0)
    === TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-2)
    === TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-1)
        sub_test.go:571: This is a t.Log from sub-0 (message-2)
    === TestName/SubTest-1
        sub_test.go:571: This is a t.Log from sub-1 (message-1)
    === TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-3)
    === TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-3)
    === TestName/SubTest-1
        sub_test.go:571: This is a t.Log from sub-1 (message-2)
        sub_test.go:571: This is a t.Log from sub-1 (message-3)
        sub_test.go:571: This is a t.Log from sub-1 (message-4)
    === TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-4)
    === TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-4)
--- PASS: TestName (0.00s)
    --- PASS: TestName/SubTest-1 (0.00s)
    --- PASS: TestName/SubTest-2 (0.00s)

Does this look right? As you can see, it's quite a bit more chatty when there's a high degree of interleaving, which could be a bummer. On the other hand, it does separate subtests better.

If so, happy to continue on that CL and figure out a real way to implement this, instead of my hacky way. 🙂 (I also figured out a better way to capture and test subtest output, so minimally I'd like to get that into the testsuite)

@rsc
Copy link
Contributor Author

@rsc rsc commented Apr 21, 2020

The indented === lines look wrong. They have no verb and they are indented.
Also, how is it even possible to be running multiple tests without a === PAUSE line?

@rsc
Copy link
Contributor Author

@rsc rsc commented Apr 21, 2020

OK, I looked at the CL test. Starting a subtest inside a kicked-off goroutine is non-standard to say the least. The important part is that the output should not require any changes to test2json. Test2json expects === lines to be unindented and contain a verb. The obvious verb here is "CONT" for continue, so that no new verb is needed either.

Instead of introducing the output redirect in package testing, I'd rather see a test script in cmd/go, testing both go test -v and go test -json. Then the diff should be very small.

That is, I think the output should be:

=== RUN   TestName
=== RUN   TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-0)
        sub_test.go:571: This is a t.Log from sub-2 (message-1)
=== RUN   TestName/SubTest-0
=== RUN   TestName/SubTest-1
=== CONT  TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-0)
=== CONT  TestName/SubTest-1
        sub_test.go:571: This is a t.Log from sub-1 (message-0)
=== CONT  TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-2)
=== CONT  TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-1)
        sub_test.go:571: This is a t.Log from sub-0 (message-2)
=== CONT TestName/SubTest-1
        sub_test.go:571: This is a t.Log from sub-1 (message-1)
=== CONT  TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-3)
=== CONT  TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-3)
=== CONT  TestName/SubTest-1
        sub_test.go:571: This is a t.Log from sub-1 (message-2)
        sub_test.go:571: This is a t.Log from sub-1 (message-3)
        sub_test.go:571: This is a t.Log from sub-1 (message-4)
=== CONT  TestName/SubTest-2
        sub_test.go:571: This is a t.Log from sub-2 (message-4)
=== CONT  TestName/SubTest-0
        sub_test.go:571: This is a t.Log from sub-0 (message-4)
--- PASS: TestName (0.00s)
    --- PASS: TestName/SubTest-1 (0.00s)
    --- PASS: TestName/SubTest-2 (0.00s)
@jadekler
Copy link
Contributor

@jadekler jadekler commented Apr 21, 2020

Starting a subtest inside a kicked-off goroutine is non-standard to say the least.

t.Parallel was blocking for some reason, so after some investigation I threw my hands up and went with the easy out to get at what I wanted. And, it was a prototype CL, so wasn't too concerned. But good to know that's unexpected for future tests!

Instead of introducing the output redirect in package testing, I'd rather see a test script in cmd/go, testing both go test -v and go test -json. Then the diff should be very small.

SG, thanks for the direction! That helps me orient.

That is, I think the output should be:

SG, thanks for the output. I'll work on updating the CL. 👍

@jadekler
Copy link
Contributor

@jadekler jadekler commented Apr 27, 2020

I've updated https://go-review.googlesource.com/c/go/+/229085:

  • I've put the test in cmd/go/testdata/script. (still need a benchmark test, and a -json test)
  • I've updated the implementation to work correctly when FAIL and PASS occurs.
  • I've removed some of the old prototype stuff.

I'm not in love with the solution, though. It seems not-great to lock, record, and unlock everywhere we print. That seems like something that's easy to forget to do in the future. I explored adding a writer that keeps track of the test name on all outgoing writes, but quickly realized it would have to interpolate each outgoing line to figure out test name (no way to pass test name through io.Writer.Write AFAICT), which doesn't seem great.

Would love thoughts if there's a better approach. (also on anything else, but particularly that)

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 28, 2020

I explored adding a writer that keeps track of the test name on all outgoing writes, but quickly realized it would have to interpolate each outgoing line to figure out test name (no way to pass test name through io.Writer.Write AFAICT), which doesn't seem great.

Don't do it as a generic Writer. Could you implement it as a method that takes a prefix, name, suffix format, and suffix args?

@jadekler
Copy link
Contributor

@jadekler jadekler commented Apr 28, 2020

Don't do it as a generic Writer

Done. I've created a non-io.Writer writer (called testPrinter).

that takes a prefix

What would a prefix look like? Something like mytest.go:17?

name

Done.

suffix format, and suffix args

I've just used fmt.Sprintf at the call site. Is that bad? Should I have Fprintf and Printf methods?

@gopherbot
Copy link

@gopherbot gopherbot commented May 4, 2020

Change https://golang.org/cl/232157 mentions this issue: testing: reformat test chatty output

@jadekler
Copy link
Contributor

@jadekler jadekler commented May 4, 2020

Change https://golang.org/cl/232157 mentions this issue: testing: reformat test chatty output

Ignore this, bad mail :)

dnephin added a commit to hashicorp/consul that referenced this issue May 6, 2020
Replace TestWriter with NewLogBuffer.

See golang/go#38458 and
golang/go#38382 (comment)

Running tests in parallel, with background goroutines, results in test
output not being associated with the correct test.

The previous solution did not address this problem because test output
could still be hidden when it was associated with a test that did not
fail.

This commit attempts a new approach. Instead of printing all the logs,
only print when a test fails. This should work well when there are a
small number of failures, but may not work well when there are many
test failures at the same time. In those cases the failures are unlikely
a result of a specific test, and the log output is likely less useful.
dnephin added a commit to hashicorp/consul that referenced this issue May 6, 2020
Replaces  #7559

See golang/go#38458 and golang/go#38382 (comment)

Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case.

The previous solution did not address this problem because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long and contained the test name twice.

This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful.

All of the logs are printed from the test goroutine, so they should be associated with the correct test.

Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly.
@odeke-em
Copy link
Member

@odeke-em odeke-em commented May 9, 2020

I’ll add the release blocker label for Go1.15.

@toothrot
Copy link
Contributor

@toothrot toothrot commented May 14, 2020

@jadekler (and anyone else): Checking in on the status of this as an issue blocking the Go 1.15 beta.

@jadekler
Copy link
Contributor

@jadekler jadekler commented May 14, 2020

@toothrot AFAICT this is ready, just waiting for a +2. I have a +1 from bcmills already.

@gopherbot gopherbot closed this in 0a90eca May 21, 2020
dnephin added a commit to hashicorp/consul that referenced this issue May 21, 2020
Replaces  #7559

See golang/go#38458 and golang/go#38382 (comment)

Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case.

The previous solution did not address this problem because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long and contained the test name twice.

This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful.

All of the logs are printed from the test goroutine, so they should be associated with the correct test.

Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly.
dnephin added a commit to hashicorp/consul that referenced this issue May 21, 2020
Replaces  #7559

Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case.

The previous solution did not address this problem because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long and contained the test name twice.

This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful.

All of the logs are printed from the test goroutine, so they should be associated with the correct test.

Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly.

Related:
golang/go#38458 (may be fixed in go1.15)
golang/go#38382 (comment)
@bcmills
Copy link
Member

@bcmills bcmills commented May 28, 2020

@gopherbot, please backport to Go 1.14: the 1.14 output format breaks go test -json for parallel subtests, and backporting the fix to the output format is likely to be more maintainable than a one-off test2json fix that is only used for the 1.14 branch.

@gopherbot
Copy link

@gopherbot gopherbot commented May 28, 2020

Backport issue(s) opened: #39308 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@bcmills
Copy link
Member

@bcmills bcmills commented May 28, 2020

@jadekler, I noticed some awkwardness remaining with this fix using gotip. The --- PASS lines for one test may interrupt the output lines of another, without emitting another === CONT line, and then as a reader it's a bit difficult to figure out which test emitted the line immediately after the --- PASS.

@jadekler
Copy link
Contributor

@jadekler jadekler commented May 28, 2020

Ack - sounds like some test cases missing. Could you show some output? Bonus: if you have a test on hand that demonstrates this, that'd be nice too. :)

@bcmills
Copy link
Member

@bcmills bcmills commented May 29, 2020

@jadekler, I'm not sure how to do it without a time.Sleep, but https://play.golang.org/p/1zo6hpethfQ reproduces it reliably for me.

$ go version
go version devel +8f4151ea Thu May 28 22:48:53 2020 +0000 linux/amd64

$ go test -v ./main_test.go
=== RUN   TestFast
=== PAUSE TestFast
=== RUN   TestSlow
=== PAUSE TestSlow
=== CONT  TestFast
=== CONT  TestSlow
    main_test.go:26: Once begun...
--- PASS: TestFast (0.00s)
    main_test.go:31: ...is half done.
--- PASS: TestSlow (0.10s)
PASS
ok      command-line-arguments  0.116s
@jadekler
Copy link
Contributor

@jadekler jadekler commented Jun 1, 2020

Thanks @bcmills. Got some free cycles today - taking a look.

gopherbot pushed a commit that referenced this issue Jun 2, 2020
This fixes an issue raised at #38458 (comment)
in which --- PASS and --- FAIL lines would not trigger --- CONT lines
of other tests.

Change-Id: I0d8cc54d682a370d0a6ea6816a11b2e462a92efe
Reviewed-on: https://go-review.googlesource.com/c/go/+/235997
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
dnephin added a commit to hashicorp/consul that referenced this issue Jun 9, 2020
Replaces  #7559

Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case.

The previous solution did not address this problem because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long and contained the test name twice.

This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful.

All of the logs are printed from the test goroutine, so they should be associated with the correct test.

Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly.

Related:
golang/go#38458 (may be fixed in go1.15)
golang/go#38382 (comment)
dnephin added a commit to hashicorp/consul that referenced this issue Jun 9, 2020
Replaces  #7559

Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case.

The previous solution did not address this problem because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long and contained the test name twice.

This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful.

All of the logs are printed from the test goroutine, so they should be associated with the correct test.

Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly.

Related:
golang/go#38458 (may be fixed in go1.15)
golang/go#38382 (comment)
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 9, 2020

Change https://golang.org/cl/241660 mentions this issue: doc/go1.15: add line for testing streaming change

gopherbot pushed a commit that referenced this issue Jul 9, 2020
Updates #37419.
Updates #38458.
Updates #24929.

Change-Id: I793bb20fa9db4432fc3a5b69956b7108e4695081
Reviewed-on: https://go-review.googlesource.com/c/go/+/241660
Run-TryBot: Jean de Klerk <deklerk@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/242058 mentions this issue: [release-branch.go1.14] testing: capture testname on --- PASS and --- FAIL lines

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/242057 mentions this issue: [release-branch.go1.14] testing: reformat test chatty output

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 10, 2020

Change https://golang.org/cl/242059 mentions this issue: [release-branch.go1.14] cmd/go: fix parallel chatty tests on solaris-amd64 builder

gopherbot pushed a commit that referenced this issue Jul 16, 2020
In #24929, we decided to stream chatty test output. It looks like,

foo_test.go:138: TestFoo/sub-1: hello from subtest 1
foo_test.go:138: TestFoo/sub-2: hello from subtest 2

In this CL, we refactor the output to be grouped by === CONT lines, preserving
the old test-file-before-log-line behavior:

=== CONT TestFoo/sub-1
    foo_test.go:138 hello from subtest 1
=== CONT TestFoo/sub-2
    foo_test.go:138 hello from subtest 2

This should remove a layer of verbosity from tests, and make it easier to group
together related lines. It also returns to a more familiar format (the
pre-streaming format), whilst still preserving the streaming feature.

Updates #38458.
Fixes #39308.

Change-Id: Iaef94c580d69cdd541b2ef055aa004f50d72d078
Reviewed-on: https://go-review.googlesource.com/c/go/+/229085
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242057
Reviewed-by: Jean de Klerk <deklerk@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Jean de Klerk <deklerk@google.com>
gopherbot pushed a commit that referenced this issue Jul 16, 2020
…amd64 builder

The parallel chatty tests added in CL 229085 fail on the
solaris-amd64-oraclerel builder, because a +NN:NN offset time zone is
used. Allow for the `+` character in the corresponding regex to fix
these tests. Also move the '-' to the end of the character class, so it
is not interpreted as the range 9-T.

Updates #38458.
For #39308.

Change-Id: Iec9ae82ba45d2490176f274f0dc6812666eae718
Reviewed-on: https://go-review.googlesource.com/c/go/+/234978
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242059
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
gopherbot pushed a commit that referenced this issue Jul 16, 2020
… FAIL lines

This fixes an issue raised at #38458 (comment)
in which --- PASS and --- FAIL lines would not trigger --- CONT lines
of other tests.

Updates #38458.
For #39308.

Change-Id: I0d8cc54d682a370d0a6ea6816a11b2e462a92efe
Reviewed-on: https://go-review.googlesource.com/c/go/+/235997
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/242058
Reviewed-by: Jean de Klerk <deklerk@google.com>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
dnephin added a commit to hashicorp/consul that referenced this issue Jul 21, 2020
Replaces #7559

Running tests in parallel, with background goroutines, results in test output not being associated with the correct test. `go test` does not make any guarantees about output from goroutines being attributed to the correct test case.

Attaching log output from background goroutines also cause data races.  If the goroutine outlives the test, it will race with the test being marked done. Previously this was noticed as a panic when logging, but with the race detector enabled it is shown as a data race.

The previous solution did not address the problem of correct test attribution because test output could still be hidden when it was associated with a test that did not fail. You would have to look at all of the log output to find the relevant lines. It also made debugging test failures more difficult because each log line was very long.

This commit attempts a new approach. Instead of printing all the logs, only print when a test fails. This should work well when there are a small number of failures, but may not work well when there are many test failures at the same time. In those cases the failures are unlikely a result of a specific test, and the log output is likely less useful.

All of the logs are printed from the test goroutine, so they should be associated with the correct test.

Also removes some test helpers that were not used, or only had a single caller. Packages which expose many functions with similar names can be difficult to use correctly.

Related:
golang/go#38458 (may be fixed in go1.15)
golang/go#38382 (comment)
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 19, 2020

Change https://golang.org/cl/249026 mentions this issue: testing: flush test summaries to stdout atomically when streaming output

gopherbot pushed a commit that referenced this issue Sep 2, 2020
While debugging #40771, I realized that the chatty printer should only
ever print to a single io.Writer (normally os.Stdout). The other
Writer implementations in the chain write to local buffers, but if we
wrote a test's output to a local buffer, then we did *not* write it to
stdout and we should not store it as the most recently logged test.

Because the chatty printer should only ever print to one place, it
shouldn't receive an io.Writer as an argument — rather, it shouldn't
be used at all for destinations other than the main output stream.

On the other hand, when we flush the output buffer to stdout in the
top-level flushToParent call, it is important that we not allow some
other test's output to intrude between the test summary header and the
remainder of the test's output. cmd/test2json doesn't know how to
parse such an intrusion, and it's confusing to humans too.

No test because I couldn't reproduce the user-reported error without
modifying the testing package. (This behavior seems to be very
sensitive to output size and/or goroutine scheduling.)

Fixes #40771
Updates #38458

Change-Id: Ic19bf1d535672b096ba1c8583a3b74aab6d6d766
Reviewed-on: https://go-review.googlesource.com/c/go/+/249026
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Sep 2, 2020

Change https://golang.org/cl/252637 mentions this issue: [release-branch.go1.15] testing: flush test summaries to stdout atomically when streaming output

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 2, 2020

Change https://golang.org/cl/252638 mentions this issue: [release-branch.go1.14] testing: flush test summaries to stdout atomically when streaming output

gopherbot pushed a commit that referenced this issue Oct 5, 2020
…cally when streaming output

While debugging #40771, I realized that the chatty printer should only
ever print to a single io.Writer (normally os.Stdout). The other
Writer implementations in the chain write to local buffers, but if we
wrote a test's output to a local buffer, then we did *not* write it to
stdout and we should not store it as the most recently logged test.

Because the chatty printer should only ever print to one place, it
shouldn't receive an io.Writer as an argument — rather, it shouldn't
be used at all for destinations other than the main output stream.

On the other hand, when we flush the output buffer to stdout in the
top-level flushToParent call, it is important that we not allow some
other test's output to intrude between the test summary header and the
remainder of the test's output. cmd/test2json doesn't know how to
parse such an intrusion, and it's confusing to humans too.

No test because I couldn't reproduce the user-reported error without
modifying the testing package. (This behavior seems to be very
sensitive to output size and/or goroutine scheduling.)

Fixes #40880
Updates #40771
Updates #38458

Change-Id: Ic19bf1d535672b096ba1c8583a3b74aab6d6d766
Reviewed-on: https://go-review.googlesource.com/c/go/+/249026
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 51c0bdc)
Reviewed-on: https://go-review.googlesource.com/c/go/+/252638
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit that referenced this issue Oct 5, 2020
…cally when streaming output

While debugging #40771, I realized that the chatty printer should only
ever print to a single io.Writer (normally os.Stdout). The other
Writer implementations in the chain write to local buffers, but if we
wrote a test's output to a local buffer, then we did *not* write it to
stdout and we should not store it as the most recently logged test.

Because the chatty printer should only ever print to one place, it
shouldn't receive an io.Writer as an argument — rather, it shouldn't
be used at all for destinations other than the main output stream.

On the other hand, when we flush the output buffer to stdout in the
top-level flushToParent call, it is important that we not allow some
other test's output to intrude between the test summary header and the
remainder of the test's output. cmd/test2json doesn't know how to
parse such an intrusion, and it's confusing to humans too.

No test because I couldn't reproduce the user-reported error without
modifying the testing package. (This behavior seems to be very
sensitive to output size and/or goroutine scheduling.)

Fixes #40881
Updates #40771
Updates #38458

Change-Id: Ic19bf1d535672b096ba1c8583a3b74aab6d6d766
Reviewed-on: https://go-review.googlesource.com/c/go/+/249026
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
(cherry picked from commit 51c0bdc)
Reviewed-on: https://go-review.googlesource.com/c/go/+/252637
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Bryan C. Mills <bcmills@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.