proposal: testing: allow examples the use of testing.T/B/F for test framework examples

[hugelgupf]

Proposal Details

Today, examples are tests presented alongside godoc as a "func main()" typically, as shown in https://go.dev/blog/examples, and is executable like that.

I'm trying to write examples for a test framework I created (https://github.com/hugelgupf/vmtest) that requires the use of testing.TB. To create an executable example in pkg.go.dev, examples have to be able to take testing.T or testing.B as an argument.

IMO, the existing Example "API" should remain as is. If an example is given a testing.T/B/F argument, though, it should be assumed to require it and be presented that way in the documentation as well, e.g.

func ExampleFunctionname(t *testing.T) {
  t.Logf("foobar")
  // Output: foobar
}

Should appear alongside Functionname in documentation as an executable snippet like this:

package main_test

import (
  "testing"

  "packagepath"
)

func TestExample(t *testing.T) {
    t.Logf("foobar")
}

[earthboundkid]

I wrote a test framework and had this same problem. I usually use example driven development, but it was sort of hard to do that for a test framework. I had to make a dummy testing.TB, and it didn't always work very well. For example, testing.TB doesn't have T.Run, so I just stub that with code that compiles but doesn't execute.

[ianlancetaylor]

This seems fairly special purpose. Examples don't have to have outputs. Should we also permit example functions to take a slog.Logger argument? It seems hard to know when to stop.

[earthboundkid]

I think the proposal should be instead to make it easier to make a testing.T/B/F for test purposes. So the code would look like

func ExampleWhatever() {
    result := testing.RunTest(func(t *testing.T) {
        // do stuff with *testing.T
    })
    fmt.Println("Test failed was", result.Failed, "because", result.Output)
    // Output: 
    // Test failed was true because error: bad potatoes
}

This could also be used in a TestFunction to test a testing framework more easily.

[dolmen]

In my proposal #74009 I have pointed that the Go Playground already allows to run Test...(*testing.T) functions, so the execution backend for x/pkgsite is already available. An Example...(*testing.T) would just have to be renamed to Test...

I think that an Example...(*testing.T) should NOT be executed like existing Example...() functions that specify the expected output, but instead like a standard test function that uses t.Fail/Error/Fatal/...

I don't see much value in supporting benchmarks (*testing.B) or fuzzing (*testing.F) in examples. If there is a need there, that would be very niche cases. Also, they are not supported by the Go Playground.

[dolmen]

Playable examples are exposed in stdlib by package go/doc.

[dolmen]

In CL 679655 I have implemented the go/doc changes that allow to expose those Example...(*testing.T) examples in pkgsite without any changes to pkgsite itself (just rebuild pkgsite with the modified go toolchain). In that proposal the example tests are standard test functions and there is no special processing of // Output: comments.

I haven't yet implemented changes to package testing that would add those Example...(*testing.T) tests in the default list of tests.

I'm waiting for feedback. Maybe this partial implementation could help this proposal (#74009) to be considered for review?

[earthboundkid]

Another reason for this is synctest.Test, which requires a *testing.T. It is annoying having to use dummy times in order to make examples reproducible. It would be nice if we could just wrap it in synctest.Test. @neild

[neild]

It'd be nice to have example tests just for documenting the testing package, if nothing else.

I like @dolmen's suggestion (#64993 (comment)) of Example...(t *testing.T) behaving like a regular Test... function for test execution purposes, but appearing in documentation as an example.

[mariusor]

I want to support @earthboundkid's sentiment regarding using Example...() tests and synctest, if I understand it correctly.

It looks like the migration from synctest.Run(func()) to synctest.Test(*testing.T, func(*testing.T)) didn't take examples into account, or, if it did and there's an alternative, it hasn't been documented properly to allow me to find it.

To support the new API I have to initialize an otherwise useless testing.T to pass to synctest in order to have Example functions that handle time. I have an example here, as taken from a toy library.