Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign uptesting: support explicitly running non-hermetic (or non-deterministic) examples #31310
Comments
This comment has been minimized.
This comment has been minimized.
One change that comes to mind is adding a testing flag, so that Edit: a slightly nicer flag would be |
This comment has been minimized.
This comment has been minimized.
You could probably about add a //build:slowexamples tag to those slow examples and then run them with go test -tag slowexamples -run "ExampleSlow.*". Would that solve your problem? |
This comment has been minimized.
This comment has been minimized.
No; hiding the examples behind a build tag has the same issues as hiding them in an |
This comment has been minimized.
This comment has been minimized.
See previously the discussion on #30595, particularly #30595 (comment). Unfortunately, unlike tests, runnable examples do not have a |
This comment has been minimized.
This comment has been minimized.
Thanks. I did see that thread, and I agree with Ian's response there. I don't think the same applies here; tests and examples are essentially different things. Generally, the users will be interested in the examples and will want to see and run them, regardless of whether they qualify as "slow" or "integration" examples. |
This comment has been minimized.
This comment has been minimized.
I think there is a big difference between “see” and “run”. I would argue that:
The connection to |
This comment has been minimized.
This comment has been minimized.
What you said makes sense to me, for the most part. The only note I'd add is that I don't care about the speed of examples per se; I've rarely encountered an example which took a lot of resources or time. I should have probably titled this issue with "non-hermetic examples" instead of "slow examples". Or perhaps even "non-deterministic examples". |
This comment has been minimized.
This comment has been minimized.
If that is the focus, then there might be some relationship to this comment from Russ in #23799 (comment)
|
This comment has been minimized.
This comment has been minimized.
Ok, I've retitled the issue to what I think is a better summary after the initial discussion. |
We can have examples which show up in the godoc but aren't runnable, and runnable examples which are run as part of
go test
.What about examples which are too slow or unreliable for
go test
, for example those needing access to the internet? I can make them non-runnable, but what if a user wants to give one a try?go test -run ExampleSlow
doesn't work, and the next best option is copy-pasting into a main func, which is awkward.The only other option, which is what I tend to do, is to stick examples as main packages elsewhere. But that's also icky; if the example packages aren't hidden,
go install ./...
will include them, and if they're hidden in another repo or in_examples
, they are hard to find.Another disadvantage of hiding the examples in a separate dir/repo is that
go test ./...
won't check if they compile fine; one has to remember to check that dir/repo too.For some context, I'm trying to collapse the https://github.com/chromedp/examples into the main repo, starting with an example_test.go file. The second and third examples are quick, but the first accesses the internet and is slow. I don't want to run it with
go test
, but I want it to show up in godoc and allow people to run it easily to try it out.I'm not labelling this as a proposal, because I don't have a specific solution in mind; this is just a description of a problem I have when trying to document packages with non-trivial examples.
/cc @acln0 @mpvl @bcmills @ianlancetaylor