testing,go/doc: no way to properly distribute non-trivial runnable examples

[mvdan]

I filed #35852 as a proposal to try to describe this problem and propose a solution, but since it was declined, I'm falling back to simply filing an issue about the problem. The fact that the solution got rejected doesn't mean that the problem doesn't exist.

--

We can have examples which show up in the godoc but aren't runnable, and runnable examples which are run as part of go test. This is enough for most use cases.

However, take the case where one has more than a handful of examples in a single package, and they each take a non-trivial amount of time to run, such as 200ms. Since they cannot run in parallel with themselves or any other test within the same package, go test can take multiple more seconds than it really needs to.

The root of the problem is that there's no way to write examples which are:

1. Part of the documentation (i.e. godoc)
2. Runnable (i.e. on godoc.org, or via go test)
3. Scalable (i.e. can be run in parallel)

Below are some workarounds I considered:

• Rewriting some of the examples as tests. Not good, because they are lost in the godoc, and hidden from the users. Loses point 1.
• Splitting the examples in multiple packages. This makes go test ./... faster, as there's built-in parallelism between packages being tested. However, this splits up godoc pages, and it's not unreasonable to have more than a handful of examples in a single package. Loses point 1.
• Make some of the examples non-runnable. Perhaps the best option today, but still not great. A runnable example is always verified by go test, and it's easier for the user to see what it does and how to run it. Loses point 2.

In the proposal above I tried exploring a couple of solutions to allow running examples in parallel. Are there other solutions we can look into?

/cc @bcmills @leitzler @rsc

[josharian]

Examples must be run serially. But perhaps examples could be run concurrently with tests?

That’d surely break someone’s tests.

But maybe that could be done manually, if testing.M had RunExamples and RunTests (and I guess you’d need RunBenchmarks). Then people could fire up two goroutines, one for examples and one for tests.

[mvdan]

Maybe - that's an interesting idea. It won't scale terribly well, but it would be far better than what we have now.

I'm not just fixated on ways to make the current way faster, by the way. That's what I tried with the proposal that got rejected. For example: maybe we should have an entirely new standard way to provide example code or packages, when those are too slow or complex to have directly as runnable examples. Being a standard is important, because you want software like godoc to show it to the user.