runtime,time: timer.Stop returns false even when no value is read from the channel

[sukunrt]

Go version

go1.23.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/sukun/.cache/go-build'
GOENV='/home/sukun/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/sukun/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/sukun/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/sukun/dev/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/sukun/dev/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.0'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/sukun/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/sukun/dev/scratch/timer/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build59502758=/tmp/go-build -gno-record-gcc-switches'

What did you do?

For the new unbuffered timers introduced in go1.23, I expected that timer.Stop would return true, if no value is read from timer channel. This is not the case, sometimes timer.Stop returns false even when no value is read from the channel and the timer wasn't stopped before. https://go.dev/play/p/t_vaPhSAkmw

In production this is breaking quic-go: quic-go/quic-go#4659
There are details in that PR about how it's breaking Kubo v0.30.0-rc2(https://github.com/ipfs/kubo/tree/v0.30.0-rc2)

What did you see happen?

timer.Stop returns false even if no value was read. And there's no corresponding value in the timer channel. This breaks existing code which relies on tracking whether it read from the timer channel or not like the one in quic-go here: https://github.com/quic-go/quic-go/blob/master/internal/utils/timer.go

What did you expect to see?

I expected it to always return true if no value was read from the channel.

Looking at the changelog: https://go-review.googlesource.com/c/go/+/568341 there's an example of correct timer usage that shouldn't be broken.

func main() {
		t := time.NewTimer(2 * time.Second)
		time.Sleep(3 * time.Second)
		if !t.Reset(2*time.Second) {
			<-t.C
		}
		<-t.C
	}
	

I believe this will break(block forever) too in some circumstances. From the code we can see why this would happen. unlockAndRun updates the timer state and releases timer lock. modify reads t.when == 0 and returns false, then unlockAndRun gets the sendLock but cancels the push to the channel.
Repro here: https://go.dev/play/p/y_PZbPlwqrM

[gabyhelp]

Related Issues and Documentation

• time: timer reset sometimes ignored, causing delayed ticks #47762 (closed)
• runtime: starvation with frequent timer resets #38119 (closed)
• proposal: time: Timer.StopAndClear method #38945 (closed)
• time: zero value of time.Timer crashes confusingly #8776 (closed)
• runtime: new timer causes racy use of timer #35394 (closed)
• runtime: differing timer behavior in goroutines between Go 1.14 and 1.15 #41606 (closed)
• time: Timer reset broken under heavy use since go1.16 timer optimizations added #47329 (closed)
• time, runtime: scheduled timer may never fire if GOMAXPROCS is reduced #45716 (closed)
• Deadlock wenn stopping stopped timer #18553 (closed)
• Fatal error: runtimer: bad p when resetting the value of a ticker pointer #45008 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cuonglm]

From Timer.Stop documentation:

// For a chan-based timer created with NewTimer(d), as of Go 1.23,
// any receive from t.C after Stop has returned is guaranteed to block
// rather than receive a stale time value from before the Stop;
// if the program has not received from t.C already and the timer is
// running, Stop is guaranteed to return true.

So even if you don't read from the channel, but the timer is not running -> Stop could return false.

[ianlancetaylor]

The text of the issue talks about timer.Stop but the example only calls timer.Reset. Is the problem with Stop or Reset?

[sukunrt]

It's with both:
The repro for Stop: https://go.dev/play/p/t_vaPhSAkmw
The repro for Reset: https://go.dev/play/p/y_PZbPlwqrM

Both the examples Reset the timer. Then they call Stop or Reset without reading from the channel. The calls should return true, but some times they return false.

[sukunrt]

I've simplified the examples:

Stop: https://go.dev/play/p/ofa64fUdVBT
Reset: https://go.dev/play/p/wWb_381RSqq

[ianlancetaylor]

Thanks. In https://go.dev/play/p/ofa64fUdVBT there is a race between running the timer and stopping the timer. The code requires that Stop return true. But Stop is permitted to return false if the timer has already expired. The documentation for Stop says "if the program has not received from t.C already and the timer is running, Stop is guaranteed to return true." In the failure case, the timer has already expired.

If I understand you correctly, you are saying that because the timer is running, and because nothing has received from the channel, then Stop must return true. But in the failing case, the timer has expired.

The code you point to in runtime/time.go is intended to handle exactly the case: a timer that has expired racing with a call to Stop. We want to guarantee that after Stop returns we will not receive a value on the channel. So even though the timer has expired, if there is a racing call to Stop, we ensure that we don't send a value on the channel. That seems like a valid interpretation of the race.

So it's not yet obvious to me that there is a bug here. We have to handle the race somehow. We want to be sure that after Stop returns there is no value on the channel. But Stop can still return false if the timer has expired, and in that case Stop can also prevent the value from being sent on the channel.

Or so it seems to me. What am I missing? Thanks.

[sukunrt]

Thanks @ianlancetaylor, @cuonglm. Some thoughts.

1. This breaks all existing code that relies on return value of timer.Stop or timer.Reset to drain from the channel. An example here: https://github.com/quic-go/quic-go/blob/master/internal/utils/timer.go

I believe this is a bug because in the changelist entry that introduced this change there is this example:

Note that a correct version of the program would be:

	func main() {
		t := time.NewTimer(2 * time.Second)
		time.Sleep(3 * time.Second)
		if !t.Reset(2*time.Second) {
			<-t.C
		}
		<-t.C
	}

This works with either semantics, by heeding t.Reset's result.
The change should not affect correct programs.

This example also suffers from the same race condition. The same argument for Stop applies to Reset. If for some reason the timer execution is delayed, t.Reset in the example might return false in go1.23 and there will be no value in the channel after t.Reset returns.

If I understand you correctly, you are saying that because the timer is running, and because nothing has received from the channel, then Stop must return true. But in the failing case, the timer has expired.

The timer only "expires" when the value is read from the channel, no?
Consider this example. We consider that this timer hasn't "expired" when Stop is called.

timer := time.NewTimer(1 * time.Millisecond)
time.Sleep(10 * time.Second)
fmt.Println(timer.Stop()) // This will surely return true

This applies to @cuonglm 's point too

So even if you don't read from the channel, but the timer is not running -> Stop could return false.

I'm uncomfortable with this definition of not running
There are three stages to executing the timer.

1. We determine the timer can be executed, now > t.when and wait for t.lock
2. Acquire t.lock and update the timer state t.when = 0, release t.lock and wait for t.stopLock
3. Acquire t.stopLock and execute t.f

If t.Stop executes after 1 => return true
If t.Stop executes after 2 before 3 => return false
If t.Stop executes after 3 => return true

Why is the timer not running when Stop executes after 2 but is running when Stop executes after 3?

3. If this is expected behavior, what's the way to Reset existing timers such that it works with both go1.22 and go1.23 and that there are no values from before the Reset in the timer channel.
   Is there a clean solution I am missing?

[ianlancetaylor]

OK, I think I see what you are getting at. In the racing case, Stop is preventing the timer from sending a value on the channel. Therefore, the timer has been stopped, and Stop should return true.

[ianlancetaylor]

@gopherbot Please open a backport to 1.23.

This bug makes it difficult or impossible to write timer code that uses Stop and Reset and works correctly for all versions of Go.

[gopherbot]

Backport issue(s) opened: #69333 (for 1.23).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

[gopherbot]

Change https://go.dev/cl/611496 mentions this issue: runtime: if stop/reset races with running timer, return correct result