Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
runtime: system experienced long pauses during GC marking #27410
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
@ysmolsky We experienced similar sized pauses during the mark phase. We tested heaps of size
We saw a lot of 5-20ms pauses (where pauses are measured as a period where no network events are recorded in the trace). The larger heaps sizes saw a greater performance impact, but it looks like that was simply because the mark phase ran for much longer.
In particular we saw huge number of Idle-GC slices in the trace. If were to guess I would say the scheduler was struggling to schedule all of the Idle-GC goroutines as well and schedule meaningful work.
If you request access to the google doc, I will grant it as soon as I can.
We saw no Mark assists in the trace. I specifically looked for them, because that was my first thought.
With the way we forced the GC to run during the trace makes assists even less likely. Because an early GC run is less likely to feel like the mutators are outpacing it and ask for assists (is my understanding).
The unusual feature of this test that the system was running on a machine with 48 cores with GOMAXPROCS unset. It looks like it was trying to use all the cores during GC, but most GC slices are idle.
The blog post has been published
If anything written there is incorrect I am happy to make edits.
Should be easier to access than the google doc.
The particular program where we saw this behaviour was a one off. We ran it for about a week it was never run again. It more or less had the whole machine to itself.
This experience was somewhat accidental, in that the service tested here was deployed without a GOMAXPROCS so it naturally just used all the cores available. So we don't experience this generally today, or we are not aware of experiencing it.
If we see any behaviour like this again I'll be sure to post new information.
From my perspective I think that this issue should probably be closed, just out of pragmatism. The version of Go is out of date and there's no practical way for us to reproduce this today.
I am personally very interested about this behaviour so we may try to reproduce it in a different system, but if we do then I can post a new issue.