Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: ReadMemStats called in a loop may prevent GC #40459

Open
mknyszek opened this issue Jul 28, 2020 · 4 comments
Open

runtime: ReadMemStats called in a loop may prevent GC #40459

mknyszek opened this issue Jul 28, 2020 · 4 comments
Assignees
Milestone

Comments

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Jul 28, 2020

In rolling forward the SetMaxHeap patches, I discovered that, as a result of the fix to #19112, if ReadMemStats is called in a tight loop, then it might keep stomping on other goroutines trying to stop the world by reacquiring worldsema before others get the chance. The worst case is if it stomps on a goroutine trying to stop the world for either starting or stopping a GC, which messes with pacing significantly.

The good news is, ReadMemStats is known to be a heavyweight operation, and so isn't called often. This is really only a problem if it's called in a tight loop. In the reproduction "tight loop" consisted of a single goroutine performing a large allocation, calling ReadMemStats, then doing some arithmetic and predictable branches. Adding in even a 1 ms sleep is enough to resolve the issue. This case is more likely to appear in tests and examples, where the behavior will be unexpected.

@mknyszek mknyszek added this to the Go1.16 milestone Jul 28, 2020
@mknyszek mknyszek self-assigned this Jul 28, 2020
@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Jul 28, 2020

CC @aclements @dr2chase

Fix is already uploaded, just need to associate it with this issue.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Jul 28, 2020

FTR: the fix is to make starting the world hand off worldsema directly to one of the waiting goroutines. This makes the behavior around the semaphore more fair, and helps prevent the situation described in the original post.

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 28, 2020

Change https://golang.org/cl/243977 mentions this issue: runtime: release worldsema with a direct G handoff

@odeke-em
Copy link
Member

@odeke-em odeke-em commented Jul 29, 2020

Thank you for reporting this @mknyszek and thank you for having the fix steady and ready! Could we perhaps include a footnote BUG in the runtime package about this, or perhaps also in the release notes as an acknowledgement of the bug?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.