Repository navigation
testing: T.Context usability concerns #18199
Description
Activity
Issue #18182 was another person asking for this last night.
Their motivation: #18182 (comment)
Copying here the point made in golang-dev, for clarity:
Such a flagging mechanism that by definition will only be useful when things will continue running in the background past the test completion does feel like a wart to me. We should be discouraging people from doing that in the first place, because arbitrary background logic delayed for arbitrary periods of time while other things are being tested is a common source of testing pain, breaking in curious and hard to reproduce ways.
As people pointed out, it's not a mere case of putting logic locally. The context, by definition, will only trigger code to start aborting after it's too late. An aborting yet running goroutine can do whatever it wants in order to attempt to stop cleanly.
In my own testing, services generally have a Stop method because of that. Internally, they leverage a tomb to flag the stopping and block until everything is done cleaning themselves up. Stop also usually returns an error, so one can also check that the stop was itself clean.
Summary of my thoughts.
We should either:
- Add both (a mechanism to signal for termination AND a mechanism to wait for all to be terminated)
- Not add anything new at all
I understand most uses of
Contextdon't care about waiting for termination, but I believe it is important enough in a significant number of testing purposes.Some arguments against the status quo (i.e., only adding
Context):- The same argument for adding
Contextfor convenience can also be made for adding some wait mechanism. AddingContextsaves 2 lines of code. Adding some mechanism for waiting saves another 2 lines. It's a slippery slope. Where do we stop? - Adding
Contextalone makes it more confusing how to add your own wait mechanism since there is no way to insert a post-cancelation wait. That is, the testing framework will only callcancelwhen theTestfunction returns, but using your ownWaitGroup.Waitwill prevent theTestfunction from even returning. - You could resolve the above issue by just not using
T.Contextand roll with your ownContextandWaitGroup, but now you have to deal with remembering whether you should be using your ownctxvariable that you pass around orT.Context. - We already have a problem with many tests calling
T's method when theTestfunction returns. The lack of a built-in waiting mechanism encourages people to do this incorrectly. Imagine the following:go func() { select { ... case <-t.Context().Done(): // Closed when Test returns if err := something.Close(); err != nil { t.Errorf("unexpected Close error: %v", err) // This is problematic! } } }()
If we go with adding a wait mechanism, I don't feel strongly whether it's by exposing a
WaitGroup, or by adding aGomethod as @bcmills suggested in #16221, or something else.There is one benefit of adding
Contextthat does not apply to a mechanism for waiting for cleanup: it is difficult to construct aContextwhich is canceled when the test has failed but the test function is still running (e.g. whent.Failed()transitions to true), but it's fairly easy todefer func() { cancel(); wg.Wait() }().So a minimal fix might be a
Contextwhich is canceled only on failure (and which otherwise explicitly leaks goroutines), or perhaps aContextwhich is canceled on failure and explicitlypanics if its methods are called after the test function has returned. That would address the "finding out when the test has failed" case but leave the "cleaning up the test" boilerplate in user code (where it is today).I think canceling the
Contextupon first failure is inconsistent with how tests work. Whent.Erroris called, it is normal to continue moving on and perform other tests. If the firstt.Errorcaused the context to be canceled, then that would affect the results of following tests within the sameTestfunction. I'm okay with cancelation occuring whent.Fatalis called, but that is included in the current behavior.The fact that people have differing expectations of when
T.Contextgets canceled is an argument against adding it. At least with rolling with your own context, the expectations are clear. So far there have been at least 3 expressed ways people expect it gets canceled:- (current behavior) Only when the
Testends (either theTestreturns ort.Fatalis called) - Only when the
Testfails - Only when the
Testtimes out (however that is defined) - Some combination of the above...
- (current behavior) Only when the
When t.Error is called, it is normal to continue moving on and perform other tests.
Good point. Still, especially given the ability to create subtests I think there are a number of situations in which "cancel on first failure" really is what you want. But perhaps that argues for something more like
errgroup, combining "cancel on first failure", "ignore subsequent errors", and "wait for cleanup" into a single API.Another data point. There are use-cases where we want to add graceful timeout to tests. The current implementation of
t.Contextprovides no way for the user to manually cancel the context other than returning from theTestfunction.If you want a timeout, you can make a derived context from t.Context(). Again, this isn't supposed to be the end-all Context for all imaginable use cases. It's supposed to be a convenience thing.
And having a context available would've helped a CL I reviewed yesterday (https://go-review.googlesource.com/c/33972/1/http2/transport_test.go) if the context were available. But it was a few lines of code without t.Context, so we survived.
But obvious you dislike it, so feel free to send a CL removing it and the documentation from go1.8.html.
Perfect is the enemy of good.
I'm uploading a CL to revert this change. There are a fair number of people who have reservations about this.
The main argument for adding
t.Contextseems to be for convenience (saving ~2 lines) and for consistency with the rest of standard library (which is moving towards addingContextwhere it makes sense). However, the benefit of convenience and consistency is offset by increased risk of users callingt.Errorafter theTesthas returned. Overall, I don't believe a 2-line convenience is worth the real risk of impropert.Errorusage (which is already a fairly common problem in tests).To be fair, I originally voted in favor of this change in #16221, but changed my position on this after trying to use it on several unit tests.
I am personally still in support adding
t.Contextin conjunction with a wait mechanism as this will counter the current detriment of addingt.Contextonly. However, I don't believe that we should rush on more API design and we can revisit this for Go 1.9.CL https://golang.org/cl/34141 mentions this issue.
One possibility might be to add:
// Go starts a new goroutine running f. The passed context is canceled // when the test completes. The next test will not be run until // f returns - the test will fail if f takes too long to return after // the test has completed. func (t *T) Go(f func(context.Context))That way the test can at least wait for the goroutine to finish
before ending the test.CL https://golang.org/cl/34242 mentions this issue.
- added a commit that references this issue
on Dec 9, 2016 CL https://golang.org/cl/34274 mentions this issue.
- added a commit that references this issue
on Dec 10, 2016 If the idea of
T.Context()is ever revisited, I'd suggest considering also including the test name with WithValue. This would mean that if you're using a logging mechanism other thanT.Logwith parallel tests (andtest -v), you'd have an easy way of labeling logs with the test name.- locked and limited conversation to collaborators
on Dec 16, 2017
https://groups.google.com/d/topic/golang-dev/rS6psxIf17Q/discussion
Decide on whether to keep/change/remove the API before the 1.8 release.