runtime: helloworld net/http asks 700MB VSS on mips32 #43699
What version of Go are you using (
The text was updated successfully, but these errors were encountered:
Also this sound suspiciously familiar. The
I just quickly did some of the math for 32-bit mips by hand in case there's some crazy overflow happening here or something, since technically we treat 32-bit mips as having a 31-bit address space.
The math seems to work out though, at least for the page allocator. It comes out to around 4680 bytes (then rounded up a physical page in size) for the summary structure, plus another 32 KiB for the page bitmap. It's still possible the page allocator is the problem, there could still be a bug or it's possible I made a mistake. Best thing would be to just run this on the right hardware with some extra print statements to understand what's going on.
Alrighty, so I dug into this and the large reservation appears to be WAI. The reservation is not from the changes in Go 1.14; this code is quite old. (And silly me, I actually knew we did this; I had to update the relevant code some time ago, but the 600 MiB total really threw me off.)
On 32-bit platforms there are two big up-front reservations. First, we attempt to make a large reservation up-front for heap bookkeeping data structures. Then we make a large reservation for the heap itself because we're concerned about fragmenting the address space (see https://cs.opensource.google/go/go/+/master:src/runtime/malloc.go;l=551;bpv=1).
Note that this reservation is strictly
So that solves the puzzle of where the VSS is coming from. But the other question is why you're running into overcommit issues. I suspect that this may be due to some heap bookkeeping data structures which are not yet used (and thus don't appear in RSS) but are committed as read-write. For instance, there's a bitmap that accounts for about 3% overhead of the peak heap usage, and this is all memory that stays committed.
But that doesn't square with some of the values I'm seeing. On a real
The total amount of committed memory is thus 5189632 bytes, or just about 5 MiB. The physical page size on this machine is 8 KiB. If the page size is higher on your machine, these numbers will likely be slightly higher (round all of them up to that size).
You mentioned in the golang-nuts thread running about 40 of these simultaneously, correct? By my calculations, the most you could run if you weren't using any other memory on the system would be 25 before overcommit fired. It's also not super surprising to me that overcommit fires first in your case because a lot of this memory (including that 4 MiB read-write mapping, which is the first heap arena) hasn't been touched. The application is otherwise sitting idle, so it's not necessarily using all the space allocated for the various data structures, and it may not have used the whole heap, either.
So, that's my best analysis of the situation.
Regarding the large VSS usage, that's WAI and a problem for
Your immediate problem can likely only be remedied by having a smaller minimum heap size (the current is 4 MiB) and mapping in the heap more incrementally. A smaller minimum heap size is blocked hard on #42430 which I'm actively investing time into resolving, but will take time to complete.