Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
runtime: potential deadlock cycle caused by scavenge.lock [1.13 backport] #34150
Just in case.
It seems we were affected by this in https://github.com/gallir/smart-relayer It was in production and we were not able to debug/trace it properly but we found that most of the connections were blocked (every one is managed by one or several goroutines) and the RSS reached the instances RAM (32 GB) and were killed by the OOM.
We went back to a version compiled with 1.12 and didn't see the issue again:
$ ps axl| grep smart-relayer
As you can see, the runtime starts tens of threads:
…cavenge.lock There are currently two edges in the lock cycle graph caused by scavenge.lock: with sched.lock and mheap_.lock. These edges appear because of the call to ready() and stack growths respectively. Furthermore, there's already an invariant in the code wherein mheap_.lock must be acquired before scavenge.lock, hence the cycle. The fix to this is to bring scavenge.lock higher in the lock cycle graph, such that sched.lock and mheap_.lock are only acquired once scavenge.lock is already held. To faciliate this change, we move scavenger waking outside of gcSetTriggerRatio such that it doesn't have to happen with the heap locked. Furthermore, we check scavenge generation numbers with the heap locked by using gopark instead of goparkunlock, and specify a function which aborts the park should there be any skew in generation count. Fixes #34150. Change-Id: I3519119214bac66375e2b1262b36ce376c820d12 Reviewed-on: https://go-review.googlesource.com/c/go/+/191977 Run-TryBot: Michael Knyszek <firstname.lastname@example.org> TryBot-Result: Gobot Gobot <email@example.com> Reviewed-by: Keith Randall <firstname.lastname@example.org> (cherry picked from commit 62e4156) Reviewed-on: https://go-review.googlesource.com/c/go/+/197501 Reviewed-by: Austin Clements <email@example.com>