Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: support running examples in parallel #35852

Open
mvdan opened this issue Nov 26, 2019 · 5 comments
Labels
Projects
Milestone

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Nov 26, 2019

Summary

Examples that run with go test always do so sequentially with all other examples and tests. I think this doesn't scale well at all with the size of packages or amount of examples, and encourages developers to only write a few simple examples per package.

I propose two solutions below; I lean towards the second one, which is more powerful.

Description

We can have examples which show up in the godoc but aren't runnable, and runnable examples which are run as part of go test. This is enough for most use cases.

However, take the case where one has more than a handful of examples in a single package, and they each take a non-trivial amount of time to run, such as 200ms. Since they cannot run in parallel with themselves or any other test within the same package, go test can take multiple seconds than it really needs to.

The root of the problem is that there's no way to do the equivalent of a test's t.Parallel() for a test.

Below are some workarounds I considered:

  • Rewriting some of the examples as tests. Not good, because they are lost in the godoc, and hidden from the users.
  • Splitting the examples in multiple packages. This makes go test ./... faster, as there's built-in parallelism between packages being tested. However, this splits up godoc pages, and it's not unreasonable to have more than a handful of examples in a single package.
  • Make some of the examples non-runnable. Perhaps the best option today, but still not great. A runnable example is always verified by go test, and it's easier for the user to see what it does and how to run it.

I propose two solutions to this.

Solution one: a variant of // Output:

A different special comment that marks the example as parallel. For example:

func ExampleFoo() {
    ...

    // Output (parallel):
    // ...
}

The syntax here is up to bikeshedding, but the idea behind the parentheses are to allow more "modes" in the future, to mirror more testing.T methods like Parallel().

Solution two: add testing.E

To mirror testing.T, offering a subset of the methods such as Parallel(). An example could optionally take it as a parameter:

func ExampleFoo(e *testing.E) {
    e.Parallel()
    ...

    // Output:
    // ...
}

If we don't want e.Parallel() to be part of the example code, we could add another special comment to specify when the example code actually starts:

func ExampleFoo(e *testing.E) {
    e.Parallel()

    // Example:
    ...

    // Output:
    // ...
}

This change is more invasive, but it's also more consistent with tests, and allows to more easily extend examples in the future. For example, we could also use this solution to solve #31310 via a testing.E.Skipf method.

We could also even fix other examples issues like #21111; that one could be done via e.Error or e.Fatal, which I'd say is significantly better than log.Fatal or panic.

testing.E could implement testing.TB if we want, but I don't think it's necessary.

I prefer this second solution, because it's more powerful, and doesn't add more syntax to the special // Output: comment.

@mvdan mvdan added the Proposal label Nov 26, 2019
@mvdan mvdan added this to the Proposal milestone Nov 26, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Nov 26, 2019

One interesting possibility for the // Output: variant is to also allow the output to be in a nondeterministic order. For example, you could consider something like

// Output (unordered):

This crops up in examples for concurrency libraries, such as in golang.org/x/sync/errgroup.ExampleGroup_pipeline. That example should be runnable, but it currently is not because the output is emitted in nondeterministic order.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 26, 2019

That already exists as // Unordered output:, I think. That's another reason why I think we shouldn't extend the current comment syntax; // Unordered output (parallel): seems a bit heavy to me.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 27, 2019

The proposal here focuses on a way to signal that an example should be run in parallel.
This misses the larger problem that examples cannot run in parallel: they write to os.Stdout.
Each example's output needs to be captured separately.
There can only be one os.Stdout at a time, therefore only one example running at a time.
See testing/run_example.go's runExample function.
This is pretty fundamental to what an example is: as soon as you add a testing.E
and require prints to some other explicit place,
the examples stop being copy-and-paste-able code.

I don't see an obvious path forward here other than write examples that run quicker.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 27, 2019

You raise a good point about os.Stdout; I hadn't realised it was being modified like that. I definitely don't want to force rewriting of existing example code.

At the same time, I find "just write faster examples" to not be a satisfactory solution here. It feels akin to "just write faster tests" instead of writing parallel tests. For example, I have a package which drives Chrome browsers; examples are self-contained, so they must start a new Chrome process and talk to it to perform some example actions. There is absolutely no way I can make those examples run faster. All the other alternatives I've listed before are similarly unsatisfactory.

Are we sure there are no other ways to run current examples in parallel? Since they modify global state by design, how about running them as separate processes, kind of like how cmd/go's test scripts exec their own test binary to run programs?

@rsc rsc changed the title Proposal: testing: support running examples in parallel proposal: testing: support running examples in parallel Dec 4, 2019
@rsc rsc added this to Incoming in Proposals Dec 4, 2019
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Dec 5, 2019

There can only be one os.Stdout at a time, therefore only one example running at a time.

The mapping of the os and fmt identifiers depends upon import statements that are necessarily outside of the Example function itself.

It's true that if the code under test prints to os.Stdout or os.Stderr itself, we cannot test that code in parallel, but in many examples the only output is generated within the Example function itself (typically as explicit calls to fmt.Print*).

So one option to preserve copy-and-pasteability might be to scope the os and/or fmt identifiers to the Example function itself:

func ExampleFoo(e *testing.E) {
	fmt := e.MockFmt()
	e.Parallel()

	// Example:
	fmt.Println(42)

	// Output:
	// 42
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Proposals
Incoming
3 participants
You can’t perform that action at this time.