Huge pages not used when malloc allocates a new heap

[estaban]

I am trying to use libhugetlbfs to have the Java Hotspot (OpenJDK11) back all JNI malloc calls by huge pages in a multi-threaded application.

I am using libhugetlbfs through LD_PRELOAD with HUGETLB_MORECORE=yes. When malloc is called, it does not call __morecore but allocates a new heap:

#0  0x00007f83ed8c95a0 in mmap64 () from /lib64/libc.so.6
#1  0x00007f83ed85b511 in new_heap () from /lib64/libc.so.6
#2  0x00007f83ed85d549 in sYSMALLOc () from /lib64/libc.so.6
#3  0x00007f83ed85e3f9 in _int_malloc () from /lib64/libc.so.6
#4  0x00007f83ed85eaac in malloc () from /lib64/libc.so.6

I am using glibc 2.12, Linux kernel 4.9 and the most up to date version of libhugetlbfs. How can I make the malloc calls use huge pages memory?

[gemorin]

glibc only calls __morecore() for the main heap since it's used as a sbrk() hook. glibc does not expose a similar hook for new heaps and calls mmap() directly so there is not much that libhugetlbfs can do....

What you really want is convince the glibc devs to add such a hook for new heaps. Good luck with that, I still have not convinced them to make __morecore() work properly: https://sourceware.org/bugzilla/show_bug.cgi?id=20646

For now, you could try to use THP for the new heaps (you might need a madvise()) but it might not be aligned on a 2MB boundary. Also there are some hacks you could try to make this work (LD_PRELOAD catching mmap() calls, remapping the memory yourself) but they are hard to get right.

[estaban]

Would modifying the mmap calls in glibc'snew_heap source code to use the MAP_HUGETLB flag, re-compiling it, and using the new libc library to run my application be a solution (through ./ld-linux.so.2 --library-path ... myapp for example)?

[gemorin]

A custom glibc is indeed a possibility. It's definitely more complicated/intrusive than a LD_PRELOAD.

But it's not as simple as just adding a MAP_HUGETLB flag to the mmap() call.

1. You have to adjust the size and alignment in new_heap(), HP allocations must be multiple of 2MB. Make sure the other functions dealing with the heap do the right thing.

2. new_heap() uses MAP_NORESERVE to avoid wasting swap. Pages are only allocated when used. That works great for 4K pages because failures to allocate these happen rarely (i.e when you run out of memory).

However HP mem allocations are a lot more likely to fail (mem fragmentation, max size of the HP pool). If at the time of the fault, the kernel cannot allocate a HP, it will generate a SIGBUS (which by default will crash your process).

If you don't use MAP_NORESERVE, you will use a lot more memory per thread than you would with 4K pages but it will not fail during faulting. However mmap() is also more likely to fail so you'd have to add a fallback to use 4K pages in that case if you don't want to abort. You would have to make sure the rest of the code can deal with the 2 possibilities.

There might be some additional issues that are not immediately obvious to me.

[skull-squadron]

@gemorin Tcmalloc in gperftools does exactly this. See TCMALLOC_MEMFS_MALLOC_PATH for how to use hugetlbfs (not libhugetlbfs) with it. No code required (except probably a wrapper script to set LD_PRELOAD) unless someone wants to link in tcmalloc to permanently replace malloc().