runtime: lock cycle between malloc and execution tracer #53979
Comments
|
Change https://go.dev/cl/418716 mentions this issue: |
|
Change https://go.dev/cl/418720 mentions this issue: |
|
Change https://go.dev/cl/418957 mentions this issue: |
|
Change https://go.dev/cl/418955 mentions this issue: |
|
Change https://go.dev/cl/418956 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Aug 4, 2022
We're missing lock edges to trace.lock that happen only rarely. Any trace event can potentially fill up a trace buffer and acquire trace.lock in order to flush the buffer, but this happens relatively rarely, so we simply haven't seen some of these lock edges that could happen. With this change, we promote "fin, notifyList < traceStackTab" to "fin, notifyList < trace" and now everything that emits trace events with a P enters the tracer lock ranks via "trace", rather than some things entering at "trace" and others at "traceStackTab". This was found by inspecting the rank graph for things that didn't make sense. Ideally we would add a mayAcquire annotation that any trace event can potentially acquire trace.lock, but there are actually cases that violate this ranking right now. This is #53979. The chance of a lock cycle is extremely low given the number of conditions that have to happen simultaneously. For #53789. Change-Id: Ic65947d27dee88d2daf639b21b2c9d37552f0ac0 Reviewed-on: https://go-review.googlesource.com/c/go/+/418716 Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What version of Go are you using (
go version)?Current HEAD (2aa473c)
Does this issue reproduce with the latest release?
Yes, though via a different code path.
What did you do?
If the execution tracer is enabled, there's a potential though rare deadlock via a rank cycle on
mheap_.lockandtrace.lock:A:
setGCPercentorsetMemoryLimitacquiremheap_.lock->gcControllerCommit->traceHeapGoal->traceEvent->traceEventLocked->traceFlush-> acquirestrace.lockB:
traceFlushacquirestrace.lock-> triggers stack growth -> stack allocator callsmheap.allocManual->mheap.allocSpan-> acquiresmheap_.lockPath "A" violates the current lock ranking. I discovered this when I added a "may acquire" annotation on
traceEvent. But I think path "B" may be the real problem. Because stack growth can happen while holdingtrace.lock, it's pretty high in the ranking (has a low rank value). But this means that anything that holds any locks further down in the ranking, like the memory allocator, can't safely create trace events.I wonder if, like
mheap.lock, we should say thattrace.lockcan only be acquired on the system stack so stack growth can never happen. I think that would push tracing down to the leaves of the rank graph, rather than it being smack in the middle./cc @mknyszek @golang/runtime
The text was updated successfully, but these errors were encountered: