context: cancelCtx exclusive lock causes extreme contention #42564
Milestone
Comments
Thank you for raising this issue. Can you please provide more information, a sample program which demonstrates this, or if that is not possible, the pprof trace you used to identify the lock contention. Thank you |
If you have 64 hardware threads, |
As a workaround, you can guard slow Err() call with a non-blocking Done() read: done := ctx.Done()
for something {
select {
case <-done:
return ctx.Err()
default:
}
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Passed a single context with cancellation to a bunch of goroutines.
These goroutines had a cold-path compute task, interlaced with calls to
context.Err()
to detect cancellation.The loop looks something like:
What did you expect to see?
A bit of a slowdown from the context check maybe?
What did you see instead?
Slightly over 50% of the CPU time was spent in
runtime.findrunnable
. ThecancelContext
struct uses async.Mutex
, and due to extreme lock contention (64 CPU threads spamming it), this was triggeringlockSlow
. From poking at pprof, it appears that about 86% of CPU time was spent in functions related to this lock acquire.I was able to work around this by adding a counter and checking it less frequently. However, I do not think that this is an intended performance degradation path. Theoretically this could be made more efficient with
sync/atomic
, although I think async.RWMutex
would still be more than sufficient.The text was updated successfully, but these errors were encountered: