runtime: dead Gs cause span fragmentation

[randall77]

Once we allocate a G, it is allocated forever. We have no mechanism to free them.
We should free dead Gs if they sit in the global free queue for long enough. Or maybe free all of them at each GC?

I noticed this while debugging #8832. The stacks for dead Gs are freed at GC time. This leads to a fragmented heap because spans for G storage and stack storage alternate in the heap. When only the stacks are freed, the resulting free spans won't coalesce because the spans for storing the Gs aren't freed.

[randall77]

@dvyukov

[rsc]

Rephrased as "dead Gs cause span fragmentation". It's not clear we can easily free dead G's - it's hard to be sure all the pointers to them have been found. I expect there is code in the runtime that depends on G's never being freed (reused to hold other types of data). I don't want to debug that.

How big a problem is the fragmentation?

[randall77]

The fragmentation can be pretty bad, the case in #8832 makes the heap into a repeated pattern of one page allocated (for Gs) and eight pages free (once used for stacks). So although the heap is ~90% free, we'd have to grow the heap for any large allocation.

That said, I don't think it is terribly common to use lots of Gs for a while and then never use them again.

I agree this may be hard. But it is something to think about if we can find a way.

[prashantv]

That said, I don't think it is terribly common to use lots of Gs for a while and then never use them again.

Most network servers tend to create a new goroutine per incoming request (E.g., the HTTP server). If there's a sudden huge spike in requests (often not intentional, but due to a misbehaving external service), it can cause a huge number of goroutines to be created, and even after the requests have been served, the process consumes a huge amount of memory. I'm assuming this is due to the fragmentation described in this issue.

As an example, we saw one process in production with a system usage of > 100GB, while the Go runtime thought it was using < 4GB.

Pprof output:

ps output:

[mandarjog]

I am seeing a similar issue I think. runtime.malg is the most prolific. It is possible that this is in response to a sudden influx of requests.

[randall77]

@mandarjog : Why do you think this is related to fragmentation?
malg allocates goroutine stacks. Maybe you just have a lot of goroutines and thus lots of stack.

This issue is about what happens after all those goroutines complete. The syndrome I would expect to see is that the heap is mostly unused, including having little memory used by stacks, but large allocations still grow the heap.