I have a golang program that makes heavy use of memory mapped files, and runs on ARM appliances where the vendor has compiled their kernel with VMSPLIT_2G. Since the golang runtime reserves 40% of the available address space up-front, my program is limited in the amount of files it can mmap for its own purposes. When the address space is exhausted, bad things happen including panics when trying to spawn threads:
a, size:=sysReserveAligned(unsafe.Pointer(p), arenaSize, heapArenaBytes)
However, it is not clear to me how I can reduce the ~258MB reservation. The code comment states "We could reserve a smaller amount of space up front if this is problem.", so I'm looking for some guidance on how to do this.
Since userspace on 32-bit linux can access a maximum of 3GB of memory (when the kernel is compiled with VMSPLIT_3G [which is the default]), it seems like at the very least the 258MB reservation could be reduced by 1/4 to ~198MB.
I think my comment was saying that it could literally just reserve less space for the heap arenas, and fall back to using fresh mmaps when mheap_.heapArenaAlloc is exhausted.
Right now, 32-bit reserves address space for all possible arenas just to avoid interleaving the memory reservations with the heap, since that would cause memory fragmentation and make it more likely that a large allocation would fail. But on 64-bit, we don't reserve any space up front for the arenas and instead just mmap them as we need them.
It's probably a good idea to still reserve some space on 32-bit for the area metadata, but it could probably be much less. We could also try to generate hint addresses that make it less likely that the heap and the heapArenas will collide. For example, we could continue to make space for all heapArenas before the initial heap hint, but just not reserve that space. Even without the reservation, the heap is likely to grow up from the hint and not interfere with the heapArenas space unless we're really tight on address space.