Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
695 lines (555 sloc) 26.4 KB

Proposal: testing: programmatic sub-test and sub-benchmark support

Author: Marcel van Lohuizen

With input from Sameer Ajmani, Austin Clements, Russ Cox, Bryan Mills, and Damien Neil.

Last updated: September 2, 2015

Discussion at https://golang.org/issue/12166.

Abstract

This proposal introduces programmatic subtest and subbenchmarks for a variety of purposes.

Background

Adding a Run methods for spawning subtests and subbenchmarks addresses a variety of features, including:

  • an easy way to select single tests and benchmarks from a table from the command line (e.g. for debugging),
  • simplify writing a collection of similar benchmarks,
  • use of Fail and friends in subtests,
  • creating subtests from an external/dynamic table source,
  • more control over scope of setup and teardown than TestMain provides,
  • more control over parallelism,
  • cleaner code, compared to many top-level functions, both for tests and benchmarks, and
  • eliminate need to prefix each error message in a test with its subtest's name.

Proposal

The proposal for tests and benchmarks are discussed separately below. A separate section explains logging and how to select subbenchmarks and subtests on the command line.

Subtests

T gets the following method:

// Run runs f as a subtest of t called name. It reports whether f succeeded.
// Run will block until all its parallel subtests have completed.
func (t *T) Run(name string, f func(t *testing.T)) bool

Several methods get further clarification on their behavior for subfunctions. Changes inbetween square brackets:

// Fail marks the function [and its calling functions] as having failed but
// continues execution.
func (c *common) Fail()

// FailNow marks the function as having failed, stops its execution
// [and aborts pending parallel subtests].
// Execution will continue at the next test or benchmark.
// FailNow must be called from the goroutine running the
// test or benchmark function, not from other goroutines
// created during the test. Calling FailNow does not stop
// those other goroutines.
func (c *common) FailNow()

// SkipNow marks the test as having been skipped, stops its execution
// [and aborts pending parallel subtests].
// ... (analoguous to FailNow)
func (c *common) SkipNow()

A NumFailed method might be useful as well.

Examples

A simple example:

tests := []struct {
     A, B int
     Sum  int
}{
    { 1, 2, 3 },
    { 1, 1, 2 },
    { 2, 1, 3 },
}
func TestSum(t *testing.T) {
    for _, tc := range tests {
        t.Run(fmt.Sprint(tc.A, "+", tc.B), func(t *testing.T) {
            if got := tc.A + tc.B; got != tc.Sum {
                t.Errorf("got %d; want %d", got, tc.Sum)
            }
        })
    }
}

Note that we write t.Errorf("got %d; want %d") instead of something like t.Errorf("%d+%d = %d; want %d"): the subtest's name already uniquely identifies the test.

Select (sub)tests from the command line using -test.run:

go test --run=TestFoo/1+2  # selects the first test in TestFoo
go test --run=TestFoo/1+   # selects tests for which A == 1 in TestFoo
go test --run=.*/1+        # for any top-level test, select subtests matching ”1+”

Skipping a subtest will not terminate subsequent tests in the calling test:

func TestFail(t *testing.T) {
    for i, tc := range tests {
        t.Run(fmt.Sprint(tc.A, "+", tc.B), func(t *testing.T) {
            if tc.A < 0 {
                t.Skip(i) // terminate test i, but proceed with test i+1.
            }
            if got := tc.A + tc.B; got != tc.Sum {
                t.Errorf("got %d; want %d", got, tc.Sum)
            }
        })
    }
}

A more concrete and realistic use case is an adaptation of bufio.TestReader:

func TestReader(t *testing.T) {
    var texts [31]string
    str := ""
    all := ""
    for i := 0; i < len(texts)-1; i++ {
        texts[i] = str + "\n"
        all += texts[i]
        str += string(i%26 + 'a')
    }
    texts[len(texts)-1] = all

    for _, readmaker := range readMakers {
        t.Run("readmaker="+readMaker.name, func(t *testing.T) {
            for _, text := range texts {
                for _, bufreader := range bufreaders {
                    for _, bufsize := range bufsizes {
                        read := readmaker.fn(strings.NewReader(text))
                        buf := NewReaderSize(read, bufsize)
                        s := bufreader.fn(buf)
                        if s != text {
                            t.Fatalf("fn=%s bufsize=%d got=%q; want=%q",
                                bufreader.name, bufsize, got, text)
                        }
                    }
                }
            }
        })
    }
}

In this example the use of t.Fatalf avoids getting a large number of similar errors for different buffer sizes. In case of failure, testing will resume with testing the next reader.

Run a subtest in parallel:

func TestParallel(t *testing.T) {
    for _, tc := range tests {
        tc := tc // Must capture the range variable.
        t.Run(tc.Name, func(t *testing.T) {
            t.Parallel()
            ...
        })
    }
}

Run teardown code after a few tests:

func TestTeardown(t *testing.T) {
    t.Run("Test1", test1)
    t.Run("Test2", test2)
    t.Run("Test3", test3)
    // teardown code.
}

Run teardown code after parallel tests:

func TestTeardownParallel(t *testing.T) {
    // By definition, this Run will not return until the parallel tests finish.
    t.Run("block", func(t *testing.T) {
        t.Run("Test1", parallelTest1)
        t.Run("Test2", parallelTest2)
        t.Run("Test3", parallelTest3)
    })
    // teardown code.
}

Test1-Test3 will run in parallel with each other, but not with any other parallel tests. This follows from the fact that Run will block until all subtests have completed and that both TestTeardownParallel and "block" are sequential.

Flags

The -test.run flag allows filtering of subtests given a prefix path where each path component is a regular expression, applying the following rules:

  • Each test has its own (sub) name.
  • Each test has a level. A top-level test (e.g. func TestFoo) has level 1, a test invoked with Run by such test has level 2, a test invoked by such a subtest level 3, and so forth.
  • A (sub)test is run if the (level-1)-th regular expression resulting from splitting -test.run by '/' matches the test's name or if this regular expression is not defined. So for a -test.run not containing a '/' the behavior is identical to the current behavior.
  • Spaces (as defined by unicode.IsSpace) are replaced by underscores.

Rules for -test.bench are analogous, except that only a non-empty -test.bench flag triggers running of benchmarks. The implementation of these flags semantics can be done later.

####Examples: Select top-level test “TestFoo” and all its subtests:

go test --run=TestFoo

Select top-level tests that contain the string “Foo” and all their subtests that contain the string “A:3”.

go test --run=Foo/A:3

Select all subtests of level 2 which name contains the string “A:1 B:2” for any top-level test:

go test --run=.*/A:1_B:2

The latter could match, for example, struct{A, B int}{1, 2} printed with %+v.

Subbenchmarks

The following method would be added to B:

// Run benchmarks f as a subbenchmark with the given name. It reports
// whether there were any failures.
//
// A subbenchmark is like any other benchmark. A benchmark that calls Run at
// least once will not be measured itself and will only run for one iteration.
func (b *B) Run(name string, f func(b *testing.B)) bool

The Benchmark function gets an additional clarification (addition between []):

// Benchmark benchmarks a single function. Useful for creating
// custom benchmarks that do not use the "go test" command.
// [
// If f calls Run, the result will be an estimate of running all its
// subbenchmarks that don't call Run in sequence in a single benchmark.]
func Benchmark(f func(b *B)) BenchmarkResult

See the Rational section for an explanation.

Example

The following code shows the use of two levels of subbenchmarks. It is based on a possible rewrite of golang.org/x/text/unicode/norm/normalize_test.go.

func BenchmarkMethod(b *testing.B) {
    for _, tt := range allMethods {
        b.Run(tt.name, func(b *testing.B) {
            for _, d := range textdata {
                fn := tt.f(NFC, []byte(d.data)) // initialize the test
                b.Run(d.name, func(b *testing.B) {
                    b.SetBytes(int64(len(d.data)))
                    for i := 0; i < b.N; i++ {
                        fn()
                    }
                })
            }
        })
    }
}

var allMethods = []struct {
    name string
    f    func(to Form, b []byte) func()
}{
    {"Transform", transform },
    {"Iter", iter },
    ...
}

func transform(f Form, b []byte) func()
func iter(f Form, b []byte) func()

var textdata = []struct { name, data string }{
    {"small_change", "No\u0308rmalization"},
    {"small_no_change", "nörmalization"},
    {"ascii", ascii},
    {"all", txt_all},
}

Note that there is some initialization code above the second Run. Because it is outside of Run, there is no need to call ResetTimer. As Run starts a new Benchmark, it is not possible to hoist the SetBytes call in a similar manner.

The output for the above benchmark, without additional logging, could look something like:

BenchmarkMethod/Transform/small_change-8      200000           668 ns/op      22.43 MB/s
BenchmarkMethod/Transform/small_no_change-8  1000000           100 ns/op     139.13 MB/s
BenchmarkMethod/Transform/ascii-8              10000         22430 ns/op     735.60 MB/s
BenchmarkMethod/Transform/all-8                 1000        128511 ns/op      43.82 MB/s
BenchmarkMethod/Iter/small_change-8           200000           701 ns/op      21.39 MB/s
BenchmarkMethod/Iter/small_no_change-8        500000           321 ns/op      43.52 MB/s
BenchmarkMethod/Iter/ascii-8                    1000        210633 ns/op      78.33 MB/s
BenchmarkMethod/Iter/all-8                      1000        235950 ns/op      23.87 MB/s
BenchmarkMethod/ToLower/small_change-8        300000           475 ns/op      31.57 MB/s
BenchmarkMethod/ToLower/small_no_change-8     500000           239 ns/op      58.44 MB/s
BenchmarkMethod/ToLower/ascii-8                  500        297486 ns/op      55.46 MB/s
BenchmarkMethod/ToLower/all-8                   1000        151722 ns/op      37.12 MB/s
BenchmarkMethod/QuickSpan/small_change-8     2000000            70.0 ns/op   214.20 MB/s
BenchmarkMethod/QuickSpan/small_no_change-8  1000000           115 ns/op     120.94 MB/s
BenchmarkMethod/QuickSpan/ascii-8               5000         25418 ns/op     649.13 MB/s
BenchmarkMethod/QuickSpan/all-8                 1000        175954 ns/op      32.01 MB/s
BenchmarkMethod/Append/small_change-8         200000           721 ns/op      20.78 MB/s
ok      golang.org/x/text/unicode/norm  5.601s

The only change in the output is the characters allowable in the Benchmark name. The output is identical in the absence of subbenchmarks. This format is compatible with tools like benchstats.

Logging

Logs for tests are printed hierarchically. Example:

--- FAIL: TestFoo  (0.03s)
    display_test.go:75: setup issue
    --- FAIL: TestFoo/{Alpha:1_Beta:1}  (0.01s)
        display_test.go:75: Foo(Beta) = 5: want 6
    --- FAIL: TestFoo/{Alpha:1_Beta:3}  (0.01s)
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
    display_test.go:75: setup issue
    --- FAIL: TestFoo/{Alpha:1_Beta:4}  (0.01s)
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
    --- FAIL: TestFoo/{Alpha:1_Beta:8}  (0.01s)
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
        display_test.go:75: Foo(Beta) = 5; want 6
    --- FAIL: TestFoo/{Alpha:1_Beta:9}  (0.03s)
        display_test.go:75: Foo(Beta) = 5; want 6

For each header, we include the full name, repeating the name of the parent. This makes it easier to identify the specific test from within the local context and obviates the need for tools to keep track of context.

For benchmarks we adopt a different strategy. For benchmarks, it is important to be able to relate the logs that might have influenced performance to the respective benchmark. This means we should ideally interleave the logs with the benchmark results.

--- BENCH: BenchmarkForm/from_NFC-8
        normalize_test.go:768: Some message
BenchmarkForm/from_NFC/canonical/to_NFC-8            10000       15914 ns/op     166.64 MB/s
--- BENCH: BenchmarkForm/from_NFC-8
        normalize_test.go:768: Some message
--- BENCH: BenchmarkForm/from_NFC/canonical-8
        normalize_test.go:776: Some message.
BenchmarkForm/from_NFC/canonical/to_NFD-8            10000       15914 ns/op     166.64 MB/s
--- BENCH: BenchmarkForm/from_NFC/canonical-8
        normalize_test.go:776: Some message.
--- BENCH: BenchmarkForm/from_NFC/canonical/to_NFD-8
        normalize_test.go:789: Some message.
        normalize_test.go:789: Some message.
        normalize_test.go:789: Some message.
BenchmarkForm/from_NFC/canonical/to_NFKC-8           10000       15170 ns/op     174.82 MB/s
--- BENCH: BenchmarkForm/from_NFC/canonical-8
        normalize_test.go:776: Some message.
BenchmarkForm/from_NFC/canonical/to_NFKD-8           10000       15881 ns/op     166.99 MB/s
--- BENCH: BenchmarkForm/from_NFC/canonical-8
        normalize_test.go:776: Some message.
BenchmarkForm/from_NFC/ext_latin/to_NFC-8             5000       30720 ns/op      52.86 MB/s
--- BENCH: BenchmarkForm/from_NFC-8
        normalize_test.go:768: Some message
BenchmarkForm/from_NFC/ext_latin/to_NFD-8             2000       71258 ns/op      22.79 MB/s
--- BENCH: BenchmarkForm/from_NFC/ext_latin/to_NFD-8
        normalize_test.go:789: Some message.
        normalize_test.go:789: Some message.
        normalize_test.go:789: Some message.
BenchmarkForm/from_NFC/ext_latin/to_NFKC-8            5000       32233 ns/op      50.38 MB/s

No bench results are printed for "parent" benchmarks.

Rationale

Alternative

One alternative to the given proposal is to define variants of tests as top-level tests or benchmarks that call helper functions. For example, the use case explained above could be written as:

func doSum(t *testing.T, a, b, sum int) {
    if got := a + b; got != sum {
        t.Errorf("got %d; want %d", got, sum)
    }
}

func TestSumA1B2(t *testing.T) { doSum(t, 1, 2, 3) }
func TestSumA1B1(t *testing.T) { doSum(t, 1, 1, 2) }
func TestSumA2B1(t *testing.T) { doSum(t, 2, 1, 3) }

This approach can work well for smaller sets, but starts to get tedious for larger sets. Some disadvantages of this approach:

  1. considerably more typing for larger test sets (less code, much larger test cases),
  2. duplication of information in test name and test values,
  3. may get unwieldy if a Cartesian product of multiple tables is used as source,
  4. doesn't work well with dynamic table sources,
  5. does not allow for the same flexibility of inserting setup and teardown code as using Run,
  6. does not allow for the same flexibility in parallelism as using Run,
  7. no ability to terminate early a subgroup of tests.

Some of these objections can be addressed by generating the test cases. It seems, though, that addressing anything beyond point 1 and 2 with generation would require more complexity than the addition of Run introduces. Overall, it seems that the benefits of the proposed addition outweigh the benefits of an approach using generation as well as expanding tests by hand.

Subtest semantics

A subtest refers to a call to Run and a test function refers to the function f passed to Run. A subtest will be like any other test. In fact, top-level tests are semantically equivalent to subtests of a single main test function.

For all subtests holds:

  1. a Fail of a subtest causes the Fail of all of its ancestor tests,
  2. a FailNow of a test also causes its uncompleted descendants to be skipped, but does not cause any of its ancestor tests to be skipped,
  3. any subtest of a test must finish within the scope of this calling test,
  4. any Parallel test function will run only after the enclosing test function returns,
  5. at most --test.parallel subtests will run concurrently at any time.

The combination of 3 and 4 means that all subtests marked as Parallel run after the enclosing test function returns but before the Run method invoking this test function returns. This corresponds to the semantics of Parallel as it exists today.

These semantics enhance consistency: a call to FailNow will always terminate the same set of subtests.

These semantics also guarantee that sequential tests are always run exclusively, while only parallel tests can run together. Also, parallel tests created by one sequentially running test will never run in parallel with parallel tests created by another sequentially running test. These simple rules allow for fairly extensive control over parallelism.

Subbenchmark semantics

The Benchmark function defines the BenchmarkResult to be the result of running all of its subbenchmarks in sequence. This is equivalent to returning N == 1 and then the sum of all values for all benchmarks, normalized to a single iteration. It may be more appropriate to use a geometric mean, but as some of the values may be zero the usage of such is somewhat problematic. The proposed definition is meaningful and the user can still compute geometric means by replacing calls to Run with calls to Benchmark if needed. The main purpose of this definition is to define some semantics to using Run in functions passed to Benchmark.

Logging

The rules for logging subtests are:

  • Each subtest maintains its own buffer to which it logs.
  • The Main test uses os.Stdout as its “buffer”.
  • When a subtest finishes, it flushes its buffer to the parent test’s buffer, prefixed with a header to identify the subtest.
  • Each subtest logs to the buffer with an indentation corresponding to its level.

These rule are consistent with the sub-test semantics presented earlier. Combined with these semantics, logs have the following properties:

  • Logs from a parallel subtests always come after the logs of its parent.
  • Logs from a parallel subtests immediately follow the output of their parent.
  • Messages logged by sequential tests will appear in chronological order in the overall test logs.
  • Each logged message is only displayed once.
  • The output is identical to the old log format absent calls to t.Run.
  • The output is identical except for non-Letter characters being allowed in names if subtests are used.

Printing hierarchically makes the relation between tests visually clear. It also avoids repeating printing some headers.

For benchmarks the priorities for logging are different. It is important to visually correlate the logs with the benchmark lines. It is also relatively rare to log a lot during benchmarking, so repeating some headers is less of an issue. The proposed logging scheme for benchmarks takes this into account.

As an alternative, we could use the same approach for benchmarks as for tests. In that case, logs would only be printed after each top-level test. For example:

BenchmarkForm/from_NFC/canonical/to_NFC-8       10000        23609 ns/op     112.33 MB/s
BenchmarkForm/from_NFC/canonical/to_NFD-8       10000        16597 ns/op     159.78 MB/s
BenchmarkForm/from_NFC/canonical/to_NFKC-8      10000        17188 ns/op     154.29 MB/s
BenchmarkForm/from_NFC/canonical/to_NFKD-8      10000        16082 ns/op     164.90 MB/s
BenchmarkForm/from_NFD/overflow/to_NFC-8          300       441589 ns/op      38.34 MB/s
BenchmarkForm/from_NFD/overflow/to_NFD-8          300       483748 ns/op      35.00 MB/s
BenchmarkForm/from_NFD/overflow/to_NFKC-8         300       467694 ns/op      36.20 MB/s
BenchmarkForm/from_NFD/overflow/to_NFKD-8         300       515475 ns/op      32.85 MB/s
--- FAIL: BenchmarkForm
    --- FAIL: BenchmarkForm/from_NFC
        normalize_test.go:768: Some failure.
        --- BENCH: BenchmarkForm/from_NFC/canonical
            normalize_test.go:776: Just a message.
            normalize_test.go:776: Just a message.
            --- BENCH: BenchmarkForm/from_NFC/canonical/to_NFD-8
                normalize_test.go:789: Some message
                normalize_test.go:789: Some message
                normalize_test.go:789: Some message
            normalize_test.go:776: Just a message.
            normalize_test.go:776: Just a message.
        normalize_test.go:768: Some failure.
        …
    --- FAIL: BenchmarkForm-8/from_NFD
        normalize_test.go:768: Some failure.
        …
        normalize_test.go:768: Some failure.
            --- BENCH: BenchmarkForm-8/from_NFD/overflow
            normalize_test.go:776: Just a message.
            normalize_test.go:776: Just a message.
            --- BENCH: BenchmarkForm-8/from_NFD/overflow/to_NFD
                normalize_test.go:789: Some message
                normalize_test.go:789: Some message
                normalize_test.go:789: Some message
            normalize_test.go:776: Just a message.
            normalize_test.go:776: Just a message.
BenchmarkMethod/Transform/small_change-8            100000        1165 ns/op      12.87 MB/s
BenchmarkMethod/Transform/small_no_change-8        1000000         103 ns/op     135.26 MB/s
…

It is still easy to see which logs influenced results (those marked BENCH), but the user will have to align the logs with the result lines to correlate the data.

Compatibility

The API changes are fully backwards compatible. It introduces several minor changes in the logs:

  • Names of tests and benchmarks may contain additional printable, non-space runes.
  • Log items for tests may be indented.

benchstats and benchcmp may need to be adapted to take into account the possibility of duplicate names.

Implementation

Most of the work would be done by the author of this proposal.

The first step consists of some minor refactorings to make the diffs for implementing T.Run and B.Run as small as possible. Subsequently, T.Run and B.Run can be implemented individually.

Although the capability for parallel subtests will be implemented in the first iteration, they will initially only be allowed for top-level tests. Once we have a good way to detect improper usage of range variables, we could open up parallelism by introducing Go or enable calling Parallel on subtests.

The aim is to have the first implementations of T.Run and B.Run in for 1.7.

Open issues

Parallelism

Using Parallel in combination with closures is prone to the “forgetting to capture a range variable” problem. We could define a Go method analogous to Run, defined as follows:

func (t *T) Go(name string, f func(t *T)) {
    t.Run(name, func(t *T) {
        t.Parallel()
        f(t)
    }
}

This suffers from the same problem, but at least would make it a) more explicit that a range variable requires capturing and b) makes it easier to detect misuse by go vet. If it is possible for go vet to detect whether t.Parallel is in the call graph of t.Run and whether the closure refers to a range variable this would be sufficient and the Go method might not be necessary.

At first we could prohibit calls to Parallel from within subtests until we decide on one of these methods or find a better solution.

Teardown

We showed how it is possible to insert teardown code after running a few parallel tests. Thought not difficult, it is a bit clumsy. We could add the following method to make this easier:

// Wait blocks until all parallel subtests have finished. It will Skip
// the current test if more than n subtests have failed. If n < 0 it will
// wait for all subtests to complete.
func (t *T) Wait(n numFailures)

The documentation in Run would have to be slightly changed to say that Run will call Wait(-1) before returning. The parallel teardown example could than be written as:

func TestTeardownParallel(t *testing.T) {
    t.Go("Test1", parallelTest1)
    t.Go("Test2", parallelTest2)
    t.Go("Test3", parallelTest3)
    t.Wait(-1)
    // teardown code.
}

This could be added later if there seems to be a need for it. The introduction of Wait would only require a minimal and backward compatible change to the sub-test semantics.

You can’t perform that action at this time.