cmd/go: add per-test timeouts

[rsc]

Tests have an overall timeout for the entire binary but no timeout for a specific test case.
You often want to limit any particular test case to a time much shorter than the overall binary.
I propose to add the concept of per-test (function) timeouts to the go command user experience as follows.

1. Each test gets a per-test timeout. The timer for a given test only ticks down when the test is running. It does not tick down when the test is blocked in t.Parallel, nor when it is blocked in t.Run running a subtest.

2. The default per-test case timeout is 1m (one minute). If the new -testtimeout flag is specified explicitly, then that sets a different default. If the -testtimeout flag is omitted but -timeout is specified explicitly, then that sets the default too. This way, if you have one really long test and use go test -timeout=30m, the per-case timeout doesn't kick in after 1 minute and kill it anyway.

3. There is a new testing.TB method SetTimeout(d time.Duration) that allows a test to set its own timeout. Calling SetTimeout does not reset the timer. If a test runs for 30 seconds and then calls t.SetTimeout(1*time.Second), it gets killed for having timed out. A timeout set this way is inherited by subtests. (They each have their own timer.)

4. When a test timeout happens, the whole process still gets killed. There's nothing we can really do about that. But the failure does say which test function timed out.

This change will help users generally, and it will also help fuzzing, because any individual invocation of a fuzz function will now have a testing.T with a 1-minute timer, providing for detection of inputs that cause infinite loops or very slow execution.

[bcmills]

When a test timeout happens, the whole process still gets killed. There's nothing we can really do about that. But the failure does say which test function timed out.

I suspect this would help (at least somewhat) with the use-cases motivating #39038, too.

[cespare]

This sounds great. Besides being an overall better experience when there's a hanging test, it may allow for simpler test code in some situations. Sometimes I've written tests which exercise concurrent code which, if it fails, is expected to hang/deadlock; then I spawn my own timer to fail the test if it hasn't completed after, say, 5s. Depending on whether the kill-the-whole-test behavior is acceptable, I could change these to simply use t.SetTimeout(5 * time.Second) instead.

[cespare]

Would there still be a 10m default -timeout?

I would prefer there not to be, since hitting that timeout without hitting a 1m per-test timeout seems like a signal that we have a giant test suite operating correctly, not that we have a hanging/broken test.

[mvdan]

So, as per my reading, -testtimeout or -timeout just set the per-test default timeout, so T.SetTimeout always takes precedence. Is that correct?

I think that's the most intuitive to the test author. If someone wants to constrain the overall run-time of tests, they'd use the global -timeout, presumably.

2. If the -testtimeout flag is omitted but -timeout is specified explicitly, then that sets the default too.

I understand the reasoning behind this, but I'm also slightly uneasy about it. Many CIs currently use lines like go test -timeout=30m ./...; that means they would get zero net benefit from this proposal, unless they noticed they'd need to change to go test -timeout=30m -testtimeout=1m ./....

4. When a test timeout happens, the whole process still gets killed. There's nothing we can really do about that.

Could you expand on the reason why we can't do better?

As a separate point; would https://pkg.go.dev/testing#T.Deadline be updated to take into account the timeout of the running test too? I assume it would be, but just clarifying since this proposal doesn't say.

Overall I really like this idea, for the reasons Bryan and Caleb mentioned too. I think my only main worry is that setting a per-test timeout with T.SetTimeout would be hard. Some machines are one or even two orders of magnitude faster than others in terms of CPU, disk, or network, especially when you take into account busy machines, like public shared CI runners.

This is not a problem for the -timeout or -testtimeout flags; they can be changed depending on the context. I can use -testtimeout=1m on my laptop, and -testtimeout=3m on the public CI, for example. However, T.SetTimeout is fairly static and can't be adjusted depending on the environment, as far as I can see.

Maybe we could also think about "testing timeout scales"; so I could run my tests in a way that all per-test timeouts are tripled if I know my machine is significantly slower than an average laptop (e.g. public CI). I think Go's builders do something similar, where very slow computers such as small ARM/RISCV boards get many timeouts multiplied to avoid benign timeouts.

[rogpeppe]

When a test timeout happens, the whole process still gets killed. There's nothing we can really do about that. But the failure does say which test function timed out.

Would the failure include all messages logged up until the timeout?

[rsc]

I understand the reasoning behind this, but I'm also slightly uneasy about it. Many CIs currently use lines like go test -timeout=30m ./...; that means they would get zero net benefit from this proposal,

Zero net benefit, but also zero breakage. It's hard to have the latter without the former.
If those same CIs have individual test cases that run for 10 minutes, I'm trying to avoid breaking them.
Or do you think it's OK to break them? (Honest question.)

If we were starting from scratch it would be much easier to say the two are separate.

[rsc]

@rogpeppe:

Would the failure include all messages logged up until the timeout?

Yes, I would hope so. I think that's possible independent of this proposal though. Right now we just do a panic but I don't see why we couldn't go around flushing output first. (Maybe it's tricky, I don't know.)

[mvdan]

Zero net benefit, but also zero breakage. It's hard to have the latter without the former.

That's fair - I don't have a better alternative in mind. I was mainly highlighting this because we should make it clear in the release notes: that any go test -timeout users will not see this benefit unless they tweak their flags.

[rsc]

Could you expand on the reason why we can't do better? [than crashing the whole process]

We have no way to forcefully stop a goroutine asynchronously. The reason we have no way is that it is unsafe: the goroutine may be holding locks, etc. There also may be other goroutines that are running as part of the test. There is no safe way to attribute which goroutines need stopping (and then also no way to safe way to stop them).

[josharian]

There is no safe way to attribute which goroutines need stopping (and then also no way to safe way to stop them).

If we had a testing.TB Context method then the tests and benchmarks could do cooperative graceful shutdown. (Examples are a bit trickier without further changes, but are also less likely to be problematic.)

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[joe-mann]

What is the benefit of having SetTimeout(d time.Duration) as well as the -testtimeout flag? Would it not suffice to have the former, but not the latter? There are a couple of issues I see with -testtimeout.

Firstly is a basic usability concern. I would suggest that applying the same per-test timeout to all tests (being run) would be unhelpful a lot of the time. However I can much more readily think of situations where you'd want to run go test ./..., but apply specific timeouts to specific tests by adding calls to SetTimeout. But there are far fewer situations where it would make sense to apply to same per-test timeout to all tests - to do so would require you to select the longest-running test contained within the specified packages and set -testtimeout accordingly. And this requires knowledge of code base that the user who runs go test may not possess.

Secondly is a more nuanced concern about how -testtimeout fits into the existing environment. Running go test ./... -testtimeout=1m may have a significant effect on the outcome of a test - i.e. it may fail when previously it passed. And this is an important detail that is detached from the test itself. Currently, if we want a test to be executed in parallel (with other like-minded tests), we can call Parallel at the top of that test. There is no -parallel flag that causes all tests to run in parallel. Given that a timeout can have similarly outcome-altering effects on a test, would it not be more consistent, and more explicit, for per-test timeouts only be set inside the test itself?

@mvdan's suggestion of a way to scale the value passed to SetTimeout seems to me a good way of making -testtimeout largely redundant, whilst also providing a mechanism to adjust the per-test timeout according to the environment in which the tests are being run:

Maybe we could also think about "testing timeout scales"; so I could run my tests in a way that all per-test timeouts are tripled if I know my machine is significantly slower than an average laptop (e.g. public CI). I think Go's builders do something similar, where very slow computers such as small ARM/RISCV boards get many timeouts multiplied to avoid benign timeouts.

[bcmills]

@josharian, no Context method is needed. Tests and benchmarks can already do cooperative graceful shutdown via the Deadline method — we just need to allow more overhead today because all of the tests end up shutting down at once when the timeout is about to expire.

For an example of how this works today in the cmd/go tests, see:

go/src/cmd/go/script_test.go

Lines 47 to 72 in 19457a5

	var (
	ctx = context.Background()
	gracePeriod = 100 * time.Millisecond
	)
	if deadline, ok := t.Deadline(); ok {
	timeout := time.Until(deadline)
	// If time allows, increase the termination grace period to 5% of the
	// remaining time.
	if gp := timeout / 20; gp > gracePeriod {
	gracePeriod = gp
	}
	// When we run commands that execute subprocesses, we want to reserve two
	// grace periods to clean up. We will send the first termination signal when
	// the context expires, then wait one grace period for the process to
	// produce whatever useful output it can (such as a stack trace). After the
	// first grace period expires, we'll escalate to os.Kill, leaving the second
	// grace period for the test function to record its output before the test
	// process itself terminates.
	timeout -= 2 * gracePeriod
	var cancel context.CancelFunc
	ctx, cancel = context.WithTimeout(ctx, timeout)
	t.Cleanup(cancel)
	}

[josharian]

@bcmills t.Deadline lets you predict when your test will time out. It doesn't provide a means to receive the information that some other test has failed (or timed out) and that as a result you should promptly shut down.

[bcmills]

Ah, I see what you mean. But I think that is more-or-less completely orthogonal to having per-test timeouts — that's more of an interaction between t.Parallel() and the existing -test.failfast flag.

[cespare]

@joe-mann to me, most of the value of this proposal comes from -testtimeout.

The way I would use it, it's mostly about improving the handling of deadlocked tests. For example, I interact with a lot of test suites which can take several minutes to run on heavily loaded CI workers but each individual test takes a few seconds at most. It's not like I would find the longest-running test and set -testtimeout to slightly higher than that; instead, I would want -testtimeout to be high enough that no reasonable test could possibly hit it without a bug (whatever that time scale happens to be for the domain). For my test suites, the default value of 1m sounds pretty good.

Then, when a test deadlocks, go test fails faster (up to 9m sooner than it would've otherwise, assuming the default -timeout 10m, or even more if you've set -timeout higher). And for really large test suites, you don't have to go through the dance of (1) start bumping up against the 10m default timeout and get some spurious failures, (2) look into whether they are "natural" timeouts or deadlocks, and (3) raise the -timeout. (That's assuming we delete the 10m default -timeout as I brought up in #48157 (comment).)

I think SetTimeout is useful for certain tests that are specifically about checking that a particular thing doesn't hang. But I think I would use SetTimeout occasionally whereas -testtimeout would help me every day.

[rsc]

Are there any remaining objections?

[mvdan]

My only remaining worry is what I outline in the last three paragraphs in #48157 (comment) - that setting absolute time.Duration timeouts for each test feels too rigid. I think built-in support for a "scale" or multiplier would be necessary, e.g. for slower machines such as CI. Otherwise we'll either end up with realistic timeouts that cause sporadic failures on CI, or inflated timeouts that reduce flakes but take too long to make a test fail when locally iterating and testing.

A couple of comments later in the thread seem to agree that this problem warrants some attention. I personally think that some sort of multiplier seems like a sane solution, if you can roughly define what the "base" speed is - such as an average laptop from two years ago that's idle. This solution seems to have been working well enough for build.golang.org, as I understand some builders use a multiplier given slower hardware.

[rsc]

@mvdan There is already an absolute duration in -timeout=, right? It seems like maybe we should have an automatic timeout scale (essentially elevating what cmd/dist does), but that could be done in a separate, orthogonal proposal. And it might not be worth doing until we have more information, since tests can of course check for environment variables and scale on their own, and people can write helpers that do that.

[cespare]

@rsc can you confirm whether this proposal includes removing the 10m default -timeout?

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[mvdan]

@rsc I guess I don't have a problem with people implementing their own scaling, but it would be a shame for each project to reinvent a slightly different wheel with different standards on what the "base timeout scale" is. I am hoping we can at least get ahead of defining what the scale of the default timeout should be. If some projects use "enough even for the slowest machine", and others use "matching a modern fast machine", then adding built-in scaling at a later time won't work with the mismatching baselines.

[rsc]

@mvdan, please feel free to file a proposal for a scaling mechanism. Thanks!

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://golang.org/cl/363134 mentions this issue: internal/fuzz: set timeout for each exec of fuzz target

[mvdan]

As a separate point; would https://pkg.go.dev/testing#T.Deadline be updated to take into account the timeout of the running test too? I assume it would be, but just clarifying since this proposal doesn't say.

I wonder if this question got lost in the conversation. I was looking at the API again, and it says:

Deadline reports the time at which the test binary will have exceeded the timeout specified by the -timeout flag.

My intuition is that we should change the docs and behavior to account for per-test timeouts, but it's worth noting that the current docs specify that it only looks at the timeout for the whole test binary.

[gopherbot]

Change https://golang.org/cl/380497 mentions this issue: internal/lsp/regtest: eliminate arbitrary timeouts

[gopherbot]

Change https://go.dev/cl/445597 mentions this issue: internal/testenv: remove RunWithTimout