proposal: testing: add `-vf` (verbose when fail) flag to `go test` command

[vladopajic]

Problem

For large golang codebase running test can produce overwhelming test results. Developers may not care about output of all tests unless test has failed.

Solution

With new option for go test command, flag -vf (verbose when fail), developer would be able to run tests, but with verbose output of tests only if it has failed. This would reduce total test result output only to bare minimum.

Open questions

1. For test functions which want to produce additional output it would be helpful to expose io.Writer in testing.T struct which could be used to collect all outputs of tests in order to conditionally print entire test outputs.

// Consider having this test case in which
// output will be always written to stdout regardless 
// if test has failed or not.
func TestFoo(t *testing.T) {
    fmt.Printf("running test")
    
    ...
}

// In order to control outputs of tests, they would have to use 
// exposed `t.Writer` which would collect outputs of tests. 
// `go test` tool would then be able to decided weather data collected by
// this `t.Writer` would need to be outputted to stdout.
func TestFoo(t *testing.T) {
    fmt.Fprintf(t.Writer, "running test")

    ...
}

// Since most codebases already use loggers, it would be quite trivial to
// create logger with `t.Writer` and pass it where necessary.
func TestFoo(t *testing.T) {
    log := logger.New().WithWriter(t.Writer)
    log.Debug("running test")

    ...
}

[apparentlymart]

I like the idea of this, but I believe the current implementation of testing.T requires knowing whether the test is "verbose" before it produces any output.

For example, the t.Log family of functions writes into the "chatty printer" only if it's non-nil when the log function is called:

go/src/testing/testing.go

Lines 1024 to 1034 in e0c6958

	if c.chatty != nil {
	if c.bench {
	// Benchmarks don't print === CONT, so we should skip the test
	// printer and just print straight to stdout.
	fmt.Print(c.decorate(s, depth+1))
	} else {
	c.chatty.Printf(c.name, "%s", c.decorate(s, depth+1))
	}
	return
	}

c.chatty seems to be set only if running in "chatty mode", which seems to be the internal name for -v.

I think what you are proposing would require a change in approach where all of the functions must buffer log information etc in memory and discard it if the test runs to completion without failing, which might be reasonable but seems quite different than the current approach where (again, assuming I'm reading the code correctly) the log functions seem to just write directly to the output stream when verbose mode is enabled.

Do you have a different idea of how this would be implemented, or is buffering in memory what you were assuming?

(If I've misunderstood how the testing.T implementation behaves in my shallow reading of it then I apologize.)

[ianlancetaylor]

It's already the case that any messages printed through t.Log and friends are printed if the test fails. This is true even though testing.Verbose() returns false. So I don't see any need for t.Writer. Also, see #22513.

So I'm not clear on what this proposal is for. Can you give a small example? Thanks.

[vladopajic]

Thanks @ianlancetaylor, you are right. I haven't noticed t.Log before, which could be utilized instead of t.Writer. Also running tests without -v seems to be already doing what was proposed here.

[miparnisari]

@ianlancetaylor is there a way of having go test -v not print test names?