init: add MALLOC_ARENA_MAX=1 to systemd

[sangaman]

This adds the MALLOC_ARENA_MAX=1 environment variable as suggested in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific to the systemd service file definition.

Without this env var, memory usage can grow significantly especially when rpcthreads is increased above its default value.

Closes #24542.

I have tested this change myself with positive results after dealing with memory consumption issues for a long time using systemd on a 8GB RAM raspi4. I figure a similar change may be desirable for the OpenRC and CentOS init files, but I don't have a way to test them and I'm not even exactly sure how environment variables should be added there (via export statements?).

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline for information on the review process.
A summary of reviews will appear here.

[willcl-ark]

I don't believe this affects the majority of users (where the two pools per core default works fine), so I don't see why we'd update the default service file like this.

We have documentation that describes what to do to reduce memory in the event that one wants to do that?

[sangaman]

I don't believe this affects the majority of users (where the two pools per core default works fine), so I don't see why we'd update the default service file like this.

We have documentation that describes what to do to reduce memory in the event that one wants to do that?

Maybe it's my fault for not digging into the docs more but I went over a year with inflated, unpredictable RAM usage and even occasional OOM crashes on a machine that I wouldn't consider particularly memory-constrained. It looks like the reason I was seeing it was due to a higher rpcthreads setting, but I'm sure I'm not the only one that has that sort of config and they may be dealing with the same issues.

My thinking is that if in fact the performance impact is "small or absent" in bitcoind then this is a sensible default for linux machines.

[Sjors]

From the doc:

However, in Bitcoin Core very little parallel allocation happens, so the impact is expected to be small or absent.

The "expected to be" could use some benchmarks before making it a default. Does the problem only happen with high rpcthreads or equally with other (potentially more common) settings like a high -dbcache or -maxmempool?

[sangaman]

Does the problem only happen with high rpcthreads or equally with other (potentially more common) settings like a high -dbcache or -maxmempool?

From my understanding/reading of MALLOC_ARENA_MAX it affects memory usage where multiple threads are concerned. rpcthreads is the only configuration option afaict that impacts the number of threads running. Increasing dbcache or maxmempool wouldn't create new threads, only increase a set limit of RAM usage, so I don't think those alone would trigger the behavior I was seeing.

[sangaman]

More relevant discussion here: #6876 (comment)

[Sjors]

@sangaman maybe -par as well? It defaults to the number of cores - 1 and is capped at 15.

[martinus]

Now that #25325 has been merged, this might be less of an issue. This pool allocator significantly reduces the number of allocations used for CCoinsMap. (by about a factor of 2000).

[sangaman]

Now that #25325 has been merged, this might be less of an issue. This pool allocator significantly reduces the number of allocations used for CCoinsMap. (by about a factor of 2000).

Thanks! I'll try that branch out and see if resolves the underlying issue without this env var, in which case I'd agree this PR is probably not necessary.

[fanquake]

Note that internally we already set M_ARENA_MAX to 1, but that is currently restricted to 32-bit systems. I would assume everyone reporting issues (#24542) is using on 64-bit. We may be at the point where we could just drop that code entirely.

[sangaman]

I'm 64 bit indeed, but wouldn't the same concern be present on 32 bit systems?

I just restarted my node with the changes from #25325 and without the MALLOC_ARENA_MAX setting, I'll give it a few days and report back on how the memory usage is going.

[DrahtBot]

There hasn't been much activity lately. What is the status here?

Finding reviewers may take time. However, if the patch is no longer relevant, please close this pull request. If the author lost interest or time to work on this, please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

[sipa]

Just because it's desirable for the purpose of running on low-memory systems does not mean it should be a default for all systemd users of bitcoind? If it's useful always, it should probably be recommended outside a doc about reducing memory usage. Alternatively, it sounds like maybe we need to document that increasing rpcthreads may increase memory usage.

[real-or-random]

I just restarted my node with the changes from #25325 and without the MALLOC_ARENA_MAX setting, I'll give it a few days and report back on how the memory usage is going.

Any insights here? Has the issue been resolved for you?

Note that https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html says:

The default value of this tunable is 0, meaning that the limit on the number of arenas is determined by the number of CPU cores online. For 32-bit systems the limit is twice the number of cores online and on 64-bit systems, it is 8 times the number of cores online.

This means what we say in https://github.com/bitcoin/bitcoin/blob/master/doc/reduce-memory.md#linux-specific is wrong (i.e., only true for 32 bits):

By default, glibc will create up to two heap arenas per core.

[achow101]

This PR does not seem to have conceptual support. Please leave a comment if you would like this to be reopened.