Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: goroutines may allocate beyond the hard heap goal #40460

Open
mknyszek opened this issue Jul 28, 2020 · 1 comment
Open

runtime: goroutines may allocate beyond the hard heap goal #40460

mknyszek opened this issue Jul 28, 2020 · 1 comment
Labels
Milestone

Comments

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Jul 28, 2020

Today, it's possible (though unlikely) that goroutines may allocate beyond the hard heap goal.

Consider the following scenario:

  1. A goroutine (G1) spins, making many large allocations in a loop.
  2. The heap is small, or there isn't actually much mark work, so the mark phase is paced to start fairly late (let's say at 95 MiB for a 100 MiB heap goal).
  3. G1 is forced to assist the GC, and in doing so accumulates a lot of credit relative to the heap size (let's say 30 MiB for a 100 MiB heap goal because it over-assists -- not unheard of).
  4. Through background work and assists, nearly all the work in the GC is done, right on schedule.
  5. Background GC goroutines try to terminate the mark phase, but something prevents them from doing so (e.g. #40459, or more work is found) many times (perhaps we get unlucky).
  6. G1, having accumulated a bunch of credit, continues allocating in parallel during this period, unimpeded.
  7. Very little time passes, but G1 successfully uses 16 MiB of its credit to generate 16 MiB of allocations, leading to a heap of 111 MiB before sweep.

We've now exceeded the hard heap goal, which is currently 1.1 * the heap goal (so 110 MiB in this example).

The fundamental problem here is that there is nothing to push back on G1 while the GC is nearly done, or after we've exceeded the soft goal. The assist credit system is the sole mechanism through which a goroutine may be blocked and forced to assist. If it still has credit, it won't even bother. This credit mechanism is not necessarily wrong, since it's what allows us to keep the assist cost down by amortizing it, but once we're in this regime where the most important thing is to finish the GC or we're in danger of exceeding the hard goal, it stands to reason that the goroutine should not be allowed to allocate. Doing so directly may have significant negative consequences, so a real solution here needs more thought.

With larger heaps this is even less likely to happen (because allocating fast enough is hard to do), but still possible.

@mknyszek
Copy link
Contributor Author

@mknyszek mknyszek commented Jul 28, 2020

@mknyszek mknyszek added this to the Backlog milestone Jul 28, 2020
@mknyszek mknyszek added the NeedsFix label Jul 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.