proposal: testing: allow examples to return an error

[rogpeppe]

Currently, all example functions must return nothing. However, it is common
for examples to call things which might possibly return errors. Printing the
error, panicking, ignoring it or calling log.Fatal is not ideal for an example where
real code would almost always just return the error. It makes the example code
less representative of the actual code and more cluttered.

I propose that example functions should be allowed to return an error,
For runnable examples, tests would fail if the example
returns a non-nil error.

[mvdan]

/cc @shurcooL @davecheney who I believe were discussing this on Twitter recently.

I believe they mentioned using something like if err != nil { /* handle err */ }.

[ianlancetaylor]

Right now a runnable example is simply one with an // Output: (or // Unordered output:) comment. So we could extend your suggestion by saying that any example that returns an error is runnable and will be run by go test. Returning a non-nil error would be reported as a test failure.

[dmitshur]

@rogpeppe I think it would be helpful towards being able to evaluate what the end result could look like if you provided some examples of such examples. There may be subtle details that will become apparent once written down, instead of us having to use our imagination.

@mvdan I use a comment to indicate that error handling should take place in contexts such as READMEs, blog posts, snippets for reading, chat. E.g.:

// in a blog post

foo, err := Bar()
if err != nil {
    // Handle error.
}
foo.Baz()

// Output: "foo"

But in real executable Go examples, I handle errors, typically with panic:

// in example_test.go

func ExampleBar() {
    foo, err := Bar()
    if err != nil {
        panic(err)
    }
    fmt.Println(foo.Baz())

    // Output: "foo"
}

One of the concerns about having func ExampleBar() error signature is that it may require a superfluous return nil at the end:

func ExampleBar() error {
    foo, err := Bar()
    if err != nil {
        return err
    }
    fmt.Println(foo.Baz())
    return nil // this line isn't helping anyone but is neccessary

    // Output: "foo"
}

Will godoc display the final return nil? Will it detract from the example at hand? If godoc hides it, that's a special case (which needs to be implemented, documented, and adds additional mental complexity for all users of Go).

Another question is, if returning error is allowed, what about returning interface{}, error? So that I can write an even more representative example:

func ExampleBar() (*Baz, error) {
    foo, err := Bar()
    if err != nil {
        return nil, err
    }
    return foo.Baz(), nil
}

But then how does // Output: fit in there? That's one type of question this proposal opens up, and I think it's worth answering.

These are just some questions that come to mind.

Finally, I think a good exercise would be to find some typical examples from the Go standard library and see how they would be improved by this proposal. Would it be a significant improvement?

[rsc]

@rogpeppe, any thoughts on @shurcooL's latest reply?

[rsc]

ping @rogpeppe

[rogpeppe]

I think that removing the final return nil is fine, as we don't show the function declaration either - we're showing a snippet from a larger function. That said, I've just realised that golang.org (but not godoc.org) shows the example as a complete function with a main function. That leads to the question of how we should show that when the Example function returns an error.

Perhaps something like this:

package main
func example() error {
    // example code here
}
func main() {
    if err := example(); err != nil {
        log.Fatal(err)
    }
 }

When not displaying a full example (as godoc.org does) we could just show the example code without any trailing return nil.

I think that allowing a more arbitrary returned value is an interesting idea but brings its own set of issues and should be considered separately.

[ianlancetaylor]

CC @dsnet

[rsc]

Printing the
error, panicking, ignoring it or calling log.Fatal is not ideal for an example where
real code would almost always just return the error. It makes the example code
less representative of the actual code and more cluttered.

I think this is a good point, but we are also talking about making error handling clearer in some future language change. It might be good to work that out before we decide what exactly to encourage in examples.

[rsc]

On hold for error handling.

[nigeltao]

The original proposal allows ExampleXxx functions to return error, instead of returning nothing. We could extend the proposal to also allow TestXxx and BenchmarkXxx functions to optionally return error. Having TestXxx return a non-nil err would implicitly call t.Error(err) on the t *testing.T and likewise for benchmarks.

Combined with the check and default-handler mechanisms suggested in the Error Handling Go 2 Draft Design, this could reduce some boilerplate.

To be clear, func TestXxx(t *testing.T) { etc } that returned no value would still be valid, and have unchanged semantics.

On the other hand, making it easier to reduce boilerplate for insufficiently useful test failures isn't necessary a good thing. Encouraging "return on first failure" also rubs against the table driven tests philosophy somewhat, where we teach that it's useful to distinguish e.g. "all test cases failed" from "every second test case failed" from "test cases failed when foobar is positive".

As per the OTOH, I'm not sure whether I actually like this idea, but I wanted to record it before I forget.

This is similar to issue #27328 but this one is for testing functions and that one is for main functions (which are somewhat similar to test examples).