New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go test -json does not print subtest results as they become available #29811

Open
rogpeppe opened this Issue Jan 18, 2019 · 3 comments

Comments

Projects
None yet
2 participants
@rogpeppe
Copy link
Contributor

rogpeppe commented Jan 18, 2019

What version of Go are you using (go version)?

$ go version
go version devel +4b3f04c63b Thu Jan 10 18:15:48 2019 +0000 linux/amd64

When I run go test -json, it prints results for top level tests when those tests complete.
But for subtests, all the results are printed only when the last sibling subtest completes.

For example, consider the following program:

package test

import (
	"fmt"
	"testing"
	"time"
)

func Test1(t *testing.T) {
	t.Parallel()
	for i := 0; i < 3; i++ {
		i := i
		t.Run(fmt.Sprintf("testx%d", i), func(t *testing.T) {
			t.Parallel()
			time.Sleep(time.Duration(3-i) * time.Second)
		})
	}
}

func Test2(t *testing.T) {
	t.Parallel()
}

this produces the following result (each line preceded by the time that line was printed):

Output of go test -json
0:00.117 {"Time":"2019-01-18T17:55:44.379734271Z","Action":"run","Package":"command-line-arguments","Test":"Test1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380063958Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"=== RUN   Test1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380075747Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"=== PAUSE Test1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380080971Z","Action":"pause","Package":"command-line-arguments","Test":"Test1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380086613Z","Action":"run","Package":"command-line-arguments","Test":"Test2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380102357Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"=== RUN   Test2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380108309Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"=== PAUSE Test2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380112668Z","Action":"pause","Package":"command-line-arguments","Test":"Test2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380118025Z","Action":"cont","Package":"command-line-arguments","Test":"Test1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380123202Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"=== CONT  Test1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380128593Z","Action":"run","Package":"command-line-arguments","Test":"Test1/testx0"}
0:00.117 {"Time":"2019-01-18T17:55:44.380139573Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"=== RUN   Test1/testx0\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380146043Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"=== PAUSE Test1/testx0\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380150741Z","Action":"pause","Package":"command-line-arguments","Test":"Test1/testx0"}
0:00.117 {"Time":"2019-01-18T17:55:44.38015585Z","Action":"run","Package":"command-line-arguments","Test":"Test1/testx1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380161348Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"=== RUN   Test1/testx1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380169686Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"=== PAUSE Test1/testx1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380174408Z","Action":"pause","Package":"command-line-arguments","Test":"Test1/testx1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380184963Z","Action":"run","Package":"command-line-arguments","Test":"Test1/testx2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380189673Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"=== RUN   Test1/testx2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380195511Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"=== PAUSE Test1/testx2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380200093Z","Action":"pause","Package":"command-line-arguments","Test":"Test1/testx2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380205261Z","Action":"cont","Package":"command-line-arguments","Test":"Test1/testx0"}
0:00.117 {"Time":"2019-01-18T17:55:44.380209759Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"=== CONT  Test1/testx0\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380215129Z","Action":"cont","Package":"command-line-arguments","Test":"Test2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380225339Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"=== CONT  Test2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380233839Z","Action":"output","Package":"command-line-arguments","Test":"Test2","Output":"--- PASS: Test2 (0.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.38023952Z","Action":"pass","Package":"command-line-arguments","Test":"Test2","Elapsed":0}
0:00.117 {"Time":"2019-01-18T17:55:44.380245912Z","Action":"cont","Package":"command-line-arguments","Test":"Test1/testx1"}
0:00.117 {"Time":"2019-01-18T17:55:44.380250653Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"=== CONT  Test1/testx1\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380256148Z","Action":"cont","Package":"command-line-arguments","Test":"Test1/testx2"}
0:00.117 {"Time":"2019-01-18T17:55:44.380266852Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"=== CONT  Test1/testx2\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380273841Z","Action":"output","Package":"command-line-arguments","Test":"Test1","Output":"--- PASS: Test1 (0.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380281107Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx2","Output":"    --- PASS: Test1/testx2 (1.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380287211Z","Action":"pass","Package":"command-line-arguments","Test":"Test1/testx2","Elapsed":1}
0:00.117 {"Time":"2019-01-18T17:55:44.380292824Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx1","Output":"    --- PASS: Test1/testx1 (2.00s)\n"}
0:00.117 {"Time":"2019-01-18T17:55:44.380298591Z","Action":"pass","Package":"command-line-arguments","Test":"Test1/testx1","Elapsed":2}
0:00.117 {"Time":"2019-01-18T17:55:44.380303308Z","Action":"output","Package":"command-line-arguments","Test":"Test1/testx0","Output":"    --- PASS: Test1/testx0 (3.00s)\n"}
0:00.118 {"Time":"2019-01-18T17:55:44.380313673Z","Action":"pass","Package":"command-line-arguments","Test":"Test1/testx0","Elapsed":3}
0:00.118 {"Time":"2019-01-18T17:55:44.380318335Z","Action":"pass","Package":"command-line-arguments","Test":"Test1","Elapsed":0}
0:00.118 {"Time":"2019-01-18T17:55:44.380323168Z","Action":"output","Package":"command-line-arguments","Output":"PASS\n"}
0:00.118 {"Time":"2019-01-18T17:55:44.380328619Z","Action":"output","Package":"command-line-arguments","Output":"ok  \tcommand-line-arguments\t(cached)\n"}
0:00.118 {"Time":"2019-01-18T17:55:44.380336036Z","Action":"pass","Package":"command-line-arguments","Elapsed":0.001}

Note that the Test2 pass result is printed as soon as it passes, but all the other results are printed at the same time.

Although this might be a necessary restriction when running go test -v because of the way test output is indented, there doesn't seem to be a need for this restriction when the output is JSON.

@rogpeppe

This comment has been minimized.

Copy link
Contributor Author

rogpeppe commented Jan 18, 2019

Note that this is painful because it's not uncommon to gather many tests under a single umbrella. As one example, cmd/go has 139 script-based subtests under TestScript. Seeing progress when there is progress can be very useful, particularly when there's a danger of a test hanging up indefinitely.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 22, 2019

This might be fixed by #24929.

I believe that the current implementation of go test -json parses the textual output of go test -v, so while the restriction is not required by the API it will be difficult to remove in practice.

@bcmills

This comment has been minimized.

Copy link
Member

bcmills commented Jan 22, 2019

Let's revisit this after #24929 lands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment