testing: allow examples with any signature

[neild]

This is an offshoot of #64993 (comment), and an alternative to #21111 and #64993.

Proposal:

We make go doc display any function named Example or ExampleXxx where Xxx does not start with a lowercase letter as an example. If the function has any input or output parameters, go doc displays just the function with no surrounding context.

The testing package will not run example functions with parameters. If an example function with parameters contains an // Output: comment, go test will produce an error.

For example, you could write any of the following examples, and they would show up in documentation in the expected places:

package examplepackage_test

func Example_returning_an_error() error {
  f, err := os.Open("testfile")
  if err != nil {
    return err
  }
  defer f.Close()
  examplepackage.F(f)
  return nil
}

func ExampleFunc_taking_a_t(t *testing.T) {
  if err := examplepackage.Func(t); err != nil {
    t.Fatal(err)
  }
}

func Example_in_and_out(a, b int) string {
  return examplepackage.AddReturningString(a, b)
}

None of the above would be run by the testing package. If you want to run them, you can add test functions which call the example functions:

func TestExample_returning_an_error(t *testing.T) {
  if err := Example_returning_an_error(); err != nil {
    t.Fatal(err)
  }
}

func TestExampleFunc_taking_a_t(t *testing.T) {
  ExampleFunc_taking_a_t(t)
}

func TestExample_in_and_out(t *testing.T) {
  for _, test := range []struct{
    a, b int
    want string
  }{
    {1, 2, "3"},
  }{
    if got, want := Example_in_and_out(test.a, test.b), test.want; got != want {
      t.Errorf("Example_in_and_out(%v, %v) = %q, want %q", test.a, test.b, got, want)
    }
  }
}

A question is how we would display these examples in go doc output. (See #21111 (comment) for current ways go doc presents examples.)

The simple option is to display the entire example func as a non-runnable example. (By "runnable", I mean runnable in your browser in the playground as for example https://pkg.go.dev/strings#example-Cut.)

A more complex possibility which I haven't thought through fully is that perhaps if an example has a paired test function (e.g., when there is an example func named ExampleFoo and a test func named TestExampleFoo) we could display the example func in a runnable playground panel where clicking "Run" executes the test.

[gabyhelp]

Related Issues

• proposal: go/doc,testing: add playable examples using a *testing.T context #74009 (closed)
• testing: support explicitly running non-hermetic (or non-deterministic) examples #31310
• proposal: testing: allow examples the use of testing.T/B/F for test framework examples #64993
• proposal: testing: support running examples in parallel #35852 (closed)
• cmd/doc: print example code and output #56073 (closed)
• go/doc: Examples includes example functions with returns #36185 (closed)
• go/doc: add fields to Example struct to indicate type/method/suffix #23864 (closed)
• proposal: testing: remove testable examples #26971 (closed)
• testing,go/doc: no way to properly distribute non-trivial runnable examples #36586

Related Code Changes

• go/doc: ignore example functions with arguments

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[apparentlymart]

I like this simplification of #64993, but I do want to still highlight what I mentioned in #64993 (comment):

The original motivation given for #64993 was to show examples for a test framework. This proposal addresses the question of how such an example could get hold of a *testing.T, but it does not offer any clear way to write a testable example of a situation where failure is expected. I expect that many useful examples of a test helper library would involve a failing test.

Combining this with something like #64993 (comment) would address my concern. That would allow a new kind of subtest that runs independently of its parent test and lets its parent inspect the result. I expect that would also be useful in regular (non-example) tests for libraries that provide testing helpers.

The parent could then arrange to pass only if the subtest failed, and could check whether the test produced the expected log output when it failed.

[neild]

a testable example of a situation where failure is expected

This doesn't provide a way to write a runnable-in-the-playground example of an expected failure. It does let you write an example, and (with some effort) you can even test the example.

You could write:

func ExampleT_Fatal(t *testing.T) {
  t.Fatal("calling fatal fails the test")
}

This would then appear in the godoc as an example for T.Fatal. (But not as one runnable in the playground.)

Testing this example is tricky, because you can't just call it directly from a test without failing that test. What you can do is write a test which executes the example from a test in a subprocess, and then verify that the subprocess failed as expected. We do this in the testing/synctest tests:
https://go.googlesource.com/go/+/refs/tags/go1.26.3/src/testing/synctest/synctest_test.go#25

func TestFatal(t *testing.T) {
	runTest(t, nil, func() {
		synctest.Test(t, func(t *testing.T) {
			t.Fatal("fatal")
		})
	}, `^--- FAIL: TestFatal.*
    synctest_test.go:.* fatal
FAIL
$`)
}

This isn't particularly ergonomic, but it's straightforward (you can steal synctest's runTest helper, which is quite small) and the use case of testing t.Fatal calls is extremely niche.

[apparentlymart]

That runTest helper in synctest seems like one reasonable way to get the effect described in #64993 (comment), so I'm assuming your intention in mentioning it was to argue that this is relatively easy to implement outside of the standard library and not common enough a need to be worth including in the standard library. Is that right?

I was orginally responding to the use-case presented in #64993 -- being able to write testable examples for a library intended to be called from another library's tests¹ -- by noting that such libraries are often focused on providing more concise ways to describe what is expected and report failure, and so meeting that use-case seems to require a way to describe an expected failure. A solution without that isn't enough to meet that use-case, in my opinion.

I don't really have a strong opinion on whether this is common enough need to be included in the standard library, though. I could accept an argument that test utility libraries are rare enough that they should just include their own custom testing helpers inline, like the one you pointed out in synctest.

This proposal was not motivated by being able to write testable examples for test helper libraries, and so we can evaluate it just by what it explicitly sets out to do: make it possible to write example functions with arbitrary signatures and have them be included in documentation. Therefore I retract my concern.

Footnotes

1. I don't personally particularly care about "runnable in the playground" for the situation I was describing, FWIW. My motivation was purely about giving maintainers of a test library the ability to use go test to make sure their examples remain functional under future maintenance, rather than allowing readers of the documentation to run those examples inline. ↩

[neild]

"Testable examples" for a function which takes a *testing.T parameter has two parts: testable and example. That is (if I follow what you mean correctly):

• an example which appears in go doc output
• the ability to exercise that example in tests to ensure it actually does what it claims

Does that sound about right?

This proposal is a possible way to satisfy the "example" part: You can write an example function which takes a *testing.T parameter, and that example will appear in the go doc-generated documentation.

My point in #79808 (comment) was to show that that it's possible to test such an example function, even when it calls t.Fatal or otherwise fails the test. Maybe that's not good enough, and we want something ergonomic rather than merely possible.

[aclements]

Proposal review discussion:

• This seems generally helpful for writing examples, and is nice and simple.
• The problem of how to test an example that's expected to fail is separable. Two possible solutions: 1) testing could provide a function like testing.Benchmark, but that runs a test and gives the caller control over what happens with the result, 2) introduce "expected failures" (discussed in proposal: testing: add a flag to detect unnecessary skips #25951)

[apparentlymart]

Over in #21111 (comment) I shared the following, which @aclements subsequently asked me to re-share here to consolidate discussion in one place.

The other concurrent related proposal is #79808, which would allow an example to return an error as long as you either don't intend the test harness to run it or you write a little wrapper test around it.

That does seem to address the concern about the example not being "copy-pastable" since it would just look like a normal function that happens to return error. The checkable rule for whether the function is expected to return an error or not would live in the separate test, since it's not part of the example.

But it does seem like it would come at the expense of not being able to use the Output: comment mechanism as a concise way to check for expected messages printed to the stdio handles. The wrapping test would presumably need to do some manual work to substitute a bytes.Buffer (or similar) in os.Stdout and os.Stderr and then restore the previous handles once the function returns, which is considerably more boilerplate than the current declarative comment.

(I added emphasis to the part of this which @aclements pointed out as relevant to this issue.)

[aclements]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— aclements for the proposal review group

[earthboundkid]

But it does seem like it would come at the expense of not being able to use the Output: comment mechanism as a concise way to check for expected messages printed to the stdio handles. The wrapping test would presumably need to do some manual work to substitute a bytes.Buffer (or similar) in os.Stdout and os.Stderr and then restore the previous handles once the function returns, which is considerably more boilerplate than the current declarative comment.

Could that be addressed by exposing the current internal mechanism as something like testing.TestExampleOutout(expectedOutput string, example func()) TestResult?