runtime: helloworld net/http asks 700MB VSS on mips32 #43699
Comments
Unfortunately I got caught up with a few things today but I'll take a look tomorrow if I can grab our MIPS builder (shouldn't be a problem, CI is pretty quiet this close to release). |
Also this sound suspiciously familiar. The I just quickly did some of the math for 32-bit mips by hand in case there's some crazy overflow happening here or something, since technically we treat 32-bit mips as having a 31-bit address space. The math seems to work out though, at least for the page allocator. It comes out to around 4680 bytes (then rounded up a physical page in size) for the summary structure, plus another 32 KiB for the page bitmap. It's still possible the page allocator is the problem, there could still be a bug or it's possible I made a mistake. Best thing would be to just run this on the right hardware with some extra print statements to understand what's going on. |
if you have some testing code with the print embedded I'm happily to run them and send you the output, I have the hardware nearby. Thanks for looking into this quickly. |
Alrighty, so I dug into this and the large reservation appears to be WAI. The reservation is not from the changes in Go 1.14; this code is quite old. (And silly me, I actually knew we did this; I had to update the relevant code some time ago, but the 600 MiB total really threw me off.) On 32-bit platforms there are two big up-front reservations. First, we attempt to make a large reservation up-front for heap bookkeeping data structures. Then we make a large reservation for the heap itself because we're concerned about fragmenting the address space (see https://cs.opensource.google/go/go/+/master:src/runtime/malloc.go;l=551;bpv=1). Note that this reservation is strictly So that solves the puzzle of where the VSS is coming from. But the other question is why you're running into overcommit issues. I suspect that this may be due to some heap bookkeeping data structures which are not yet used (and thus don't appear in RSS) but are committed as read-write. For instance, there's a bitmap that accounts for about 3% overhead of the peak heap usage, and this is all memory that stays committed. But that doesn't square with some of the values I'm seeing. On a real
The total amount of committed memory is thus 5189632 bytes, or just about 5 MiB. The physical page size on this machine is 8 KiB. If the page size is higher on your machine, these numbers will likely be slightly higher (round all of them up to that size). You mentioned in the golang-nuts thread running about 40 of these simultaneously, correct? By my calculations, the most you could run if you weren't using any other memory on the system would be 25 before overcommit fired. It's also not super surprising to me that overcommit fires first in your case because a lot of this memory (including that 4 MiB read-write mapping, which is the first heap arena) hasn't been touched. The application is otherwise sitting idle, so it's not necessarily using all the space allocated for the various data structures, and it may not have used the whole heap, either. So, that's my best analysis of the situation. Regarding the large VSS usage, that's WAI and a problem for Your immediate problem can likely only be remedied by having a smaller minimum heap size (the current is 4 MiB) and mapping in the heap more incrementally. A smaller minimum heap size is blocked hard on #42430 which I'm actively investing time into resolving, but will take time to complete. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Not sure
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
I build a helloworld net/http using :
GOOS=linux GOARCH=mips GOMIPS=softfloat go build -ldflags="-s -w"
and run it on a mips32 board, use top to notice its VSS is 700MB and RSS is 4MB. run multiple of them in parallel will cause 'out of memory' when I set overcommit_memory to 2 very quickly. I have 128MB RAMThe code I use:
What did you expect to see?
I would expect a much smaller VSS on a 128MB 32-bit system, to compare my C/C++ http process only need about 4MB for VSS and 1MB for RSS
What did you see instead?
700MB VSS and easily 'out of memory' errors.
The text was updated successfully, but these errors were encountered: