Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: high-percentile latency of memory allocations has regressed significantly #31678

Open
mknyszek opened this issue Apr 25, 2019 · 3 comments

Comments

@mknyszek
Copy link
Contributor

commented Apr 25, 2019

Heavily-allocating and highly-parallel applications' memory allocation latencies regressed significantly in the Go 1.12 release (up to 3-4x in k8s' 99th percentile latency).

Much of the background and work done for this issue has been summarized in kubernetes/kubernetes#75833.

To re-summarize:

The offending commit with bisection was found to be 8e093e7, which introduced the notion of scavenging to make up for the RSS increase for scavenging from scavenged memory (a mouthful). This change landed to fix an issue for some users where, as a result of previous changes in Go 1.12, the allocator could outrun things like debug.FreeOSMemory.

However, these extra syscalls caused an unacceptable increase in the latency of memory allocations which took this slow path which had the heap lock acquired. In the case of an already heavily-allocating and highly-parallel application (like the k8s situation: hundreds of thousands of goroutines and 64 Ps), this can have serious consequences in terms latency because subsequent span allocations' would end up spending a lot of time trying to acquire the heap lock.

While the fundamental issue is more the fact that the span allocator is not very scalable (all span allocations are serialized via a single lock), this is much more difficult to fix. At the very least we should bring back the latency to Go 1.11 or better in these situations.

In investigating this further, we discovered that the fact that the allocator could outrun things like debug.FreeOSMemory was partially because of the VSS growth introduced by the regression in #31616 in Go 1.12, so fixing this bug could be as simple as reverting 8e093e7 and fixing #31616.

@mknyszek mknyszek added this to the Go1.13 milestone Apr 25, 2019

@mknyszek mknyszek self-assigned this Apr 25, 2019

@mknyszek

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2019

@gopherbot please backport open a backport to 1.12. I think it makes sense to fix this issue for Go 1.12 since it would at least let users safely move to Go 1.12 without having to skip a release due to performance issues.

We should do more in the future to ensure that performance regressions like this get discovered prior to release as opposed to after, perhaps through improved benchmarks.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 25, 2019

Backport issue(s) opened: #31679 (for 1.12).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 26, 2019

Change https://golang.org/cl/174101 mentions this issue: Revert "runtime: scavenge memory upon allocating from scavenged memory"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.