testing: stream t.Log output as it happens

[brikis98]

As suggested in #23213 (comment), I'm filing a new issue with a proposal for how to improve t.Log and t.Logf.

What version of Go are you using (go version)?

go version go1.9.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

I'm on Darwin, amd64, but this affects all systems/architectures.

Motivation

Consider the following (silly) automated tests:

func TestFoo(t *testing.T) {
	t.Parallel()

	for i := 0; i < 15; i++ {
		t.Logf("%d", i)
		time.Sleep(3 * time.Second)
	}
}

func TestBar(t *testing.T) {
	t.Parallel()

	for i := 0; i < 15; i++ {
		t.Logf("%d", i)
		time.Sleep(2 * time.Second)
	}
}

func TestBaz(t *testing.T) {
	t.Parallel()

	for i := 0; i < 15; i++ {
		t.Logf("%d", i)
		time.Sleep(1 * time.Second)
	}
}

If I run go test -v, I get no log output until all of TestFoo is done, then no output until all of TestBar is done, and again no more output until all of TestBaz is done. This is fine if the tests are working, but if there is some sort of bug, there are a few cases where buffering log output is problematic:

• When iterating locally, I want to be able to make a change, run my tests, see what's happening in the logs immediately to understand what's going on, hit CTRL+C to shut the test down early if necessary, make another change, re-run the tests, and so on. If TestFoo is slow (e.g., it's an integration test), I get no log output until the very end of the test. This significantly slows down iteration.
• If TestFoo has a bug that causes it to hang and never complete, I'd get no log output whatsoever. In these cases, t.Log and t.Logf are of no use at all. This makes debugging very difficult.
• Moreover, not only do I get no log output, but if the test hangs too long, either the Go test timeout kills the test after 10 minutes, or if I increase that timeout, many CI servers will also kill off tests if there is no log output after a certain amount of time (e.g., 10 minutes in CircleCI). So now my tests are killed and I have nothing in the logs to tell me what happened.

Current workarounds

The main workaround is to replace all t.Logf calls with either fmt.Printf or a custom logger. Since we often want to iterate quickly on tests, and since debugging is such an important use case, in practice, that means we never use t.Logf at all.

This seems like a missed opportunity. Go has a lot of powerful testing tools built in, and we usually pass the t variable around to many test methods so they can call t.Fatal(), t.Name(), etc. It would be great if t had a logger built in that we could actually use. Honestly, the only thing it's really missing is some way to configure the logger to write to stdout immediately, rather than buffering.

Possible solutions

Probably the easiest way to improve t.Log and t.Logf is to add a -test.streamlogs option to the go test command. When this option is set, t.Log and t.Logf write to stdout immediately, instead of buffering.

This means the output format with this option set wouldn't be backwards compatible, but the expectation is that this option would only be used for local debugging, and not something used in your CI system.

[randall77]

I think we can modify -v to get what you want. For instance:

--- a/src/testing/testing.go
+++ b/src/testing/testing.go
@@ -560,6 +560,10 @@ func (c *common) FailNow() {
 func (c *common) log(s string) {
        c.mu.Lock()
        defer c.mu.Unlock()
+       if c.chatty {
+               fmt.Printf("%s: %s", c.name, c.decorate(s))
+               return
+       }
        c.output = append(c.output, c.decorate(s)...)
 }

This will print output immediately, and you get something like this:

$ go test -v
=== RUN   TestFoo
=== PAUSE TestFoo
=== RUN   TestBar
=== PAUSE TestBar
=== RUN   TestBaz
=== PAUSE TestBaz
=== CONT  TestFoo
TestFoo: 	test_test.go:12: 0
=== CONT  TestBaz
TestBaz: 	test_test.go:30: 0
=== CONT  TestBar
TestBar: 	test_test.go:21: 0
TestBaz: 	test_test.go:30: 1
TestBar: 	test_test.go:21: 1
TestBaz: 	test_test.go:30: 2
TestFoo: 	test_test.go:12: 1
TestBaz: 	test_test.go:30: 3
TestBar: 	test_test.go:21: 2
TestBaz: 	test_test.go:30: 4
TestBaz: 	test_test.go:30: 5
TestBar: 	test_test.go:21: 3
TestFoo: 	test_test.go:12: 2
TestBaz: 	test_test.go:30: 6
TestBaz: 	test_test.go:30: 7
TestBar: 	test_test.go:21: 4
TestBaz: 	test_test.go:30: 8
TestFoo: 	test_test.go:12: 3
TestBaz: 	test_test.go:30: 9
TestBar: 	test_test.go:21: 5
TestBaz: 	test_test.go:30: 10
TestBaz: 	test_test.go:30: 11
TestBar: 	test_test.go:21: 6
TestFoo: 	test_test.go:12: 4
TestBaz: 	test_test.go:30: 12
TestBaz: 	test_test.go:30: 13
TestBar: 	test_test.go:21: 7
TestBaz: 	test_test.go:30: 14
TestFoo: 	test_test.go:12: 5
--- PASS: TestBaz (15.01s)
TestBar: 	test_test.go:21: 8
TestBar: 	test_test.go:21: 9
TestFoo: 	test_test.go:12: 6
TestBar: 	test_test.go:21: 10
TestFoo: 	test_test.go:12: 7
TestBar: 	test_test.go:21: 11
TestFoo: 	test_test.go:12: 8
TestBar: 	test_test.go:21: 12
TestBar: 	test_test.go:21: 13
TestFoo: 	test_test.go:12: 9
TestBar: 	test_test.go:21: 14
TestFoo: 	test_test.go:12: 10
--- PASS: TestBar (30.01s)
TestFoo: 	test_test.go:12: 11
TestFoo: 	test_test.go:12: 12
TestFoo: 	test_test.go:12: 13
TestFoo: 	test_test.go:12: 14
--- PASS: TestFoo (45.02s)
PASS
ok  	_/Users/khr/gowork/issue24929	45.019s

It's pretty jumbled, but gets you what you want.
I don't think there's any issue with changing the format of -v output.

[brikis98]

That would be great 👍

I agree the output is jumbled, but it's actually surprisingly easy to follow so long as each line is prefixed with the test name, and if it's very grep-friendly to make things even easier to read.

[agnivade]

Yea, this is totally OK. Usually, I run that single test when debugging a test. So I don't need to bother about other outputs getting jumbled. I just want the log output to appear immediately.

[rsc]

I'm skeptical that we can modify -v output, for what little that's worth.

[brikis98]

One more note on the topic of log streaming. It seems that if you run go test against multiple packages (e.g., go test -v ./...), the log output is buffered, and using fmt.Println or any other utility that writes to stdout does not help in this instance. I'm not sure on the buffering logic, but it seems like no log output shows up until all the tests in an entire package are done, and the log output is displayed one package at a time.

This leads to all the same issues mentioned above, and is especially problematic if you have a single slow package (e.g., one with integration tests), as many CI systems will kill your tests after a certain timeout with no log output (e.g., 10 min in CircleCI). Changing your code doesn't help in this case. The only workaround I've found is to set -p 1, so each package is tested sequentially. This seems to allow fmt.Println to stream to stdout as normal, but significantly reduces the parallelism you can get in your tests.

[ianlancetaylor]

We could argue that the output of -test.v is intended for human consumption, and as such is OK to change along the lines that @randall77 suggests above. Programs that want to parse test output should use -test.json. Of course, this means that we might need to modify the -test.json output in some way so that the test2json program can parse the new output.

[rsc]

The redefinition of -v is a bit concerning to me but maybe not enough to sink the idea.
Let's land this early in the Go 1.12 cycle, along with changes to test2json to parse the new prefixes.
If it turns out to be bad, we'll have time to roll it back.