Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
util: Work around (virtual) memory exhaustion on 32-bit w/ glibc #10120
glibc-specific: On 32-bit systems set the number of arenas to 1. By default, since glibc 2.10, the C library will create up to two heap arenas per core. This is known to cause excessive virtual address space usage in our usage. Work around it by setting the maximum number of arenas to 1.
I documented this before in Reducing bitcoind memory usage but with the reported OOM issues in 0.14 it seems more pressing to do it by default when there is low virtual memory space (e.g. on 32 bit).
This will show an entry per heap.
Tested on 32-bit ARM, 4-core:
Tested on 64-bit x86, 6-core:
<heap nr="0"> <heap nr="1"> <heap nr="2"> <heap nr="3"> <heap nr="4"> <heap nr="5"> <heap nr="6"> <heap nr="7"> <heap nr="8"> <heap nr="9"> <heap nr="10"> <heap nr="11"> <heap nr="12">
I'm not familiar with the intricacies here, so I'll defer judgement on the root issue.
The implementation suggests that it's meant to apply to glibc only, but there's no obvious check for that. Is M_ARENA_MAX itself glibc specific, or do we need some macro to check for it?
The root issue is that freed object stay around for a while for reuse, so having 8 heaps means that they can potentially stick around 8 times. This is extremely bad considering e.g. how large the consecutive objects are that leveldb allocates.
As always, feel free to suggest a better way to do the autoconf stuff. Yes, M_ARENA_MAX is specific to glibc (or anything deriving from it).