Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building requires >1GB memory #6658

Closed
TheBlueMatt opened this issue Sep 10, 2015 · 57 comments

Comments

@TheBlueMatt
Copy link
Contributor

@TheBlueMatt TheBlueMatt commented Sep 10, 2015

CXX test/test_test_bitcoin-script_tests.o
g++: internal compiler error: Killed (program cc1plus)

script_tests requires more than 1GB memory to build, killing a lot of VMs.

@theuni

This comment has been minimized.

Copy link
Member

@theuni theuni commented Sep 11, 2015

I don't think an ICE is really our concern. I assume clang manages ok?

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Sep 11, 2015

Hmm? I'd say it being impossible to build Bitcoin Core by default on many VPSes is our concern.

On September 11, 2015 12:24:16 AM EDT, Cory Fields notifications@github.com wrote:

I don't think an ICE is really our concern. I assume clang manages ok?


Reply to this email directly or view it on GitHub:
#6658 (comment)

@sipa

This comment has been minimized.

Copy link
Member

@sipa sipa commented Sep 11, 2015

@luke-jr

This comment has been minimized.

Copy link
Member

@luke-jr luke-jr commented Sep 11, 2015

@TheBlueMatt Try using CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768"

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Sep 11, 2015

The ICE in question is OOM killer eating g++.

On September 11, 2015 12:29:35 AM EDT, Pieter Wuille notifications@github.com wrote:

Internal Compiler Errot


Reply to this email directly or view it on GitHub:
#6658 (comment)

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Sep 11, 2015

@luke-jr: I can get it to build fine (--disable-tests), I just care that we fix the actual issue.

On September 11, 2015 12:31:36 AM EDT, Luke-Jr notifications@github.com wrote:

@TheBlueMatt Try using CXXFLAGS="--param ggc-min-expand=1 --param
ggc-min-heapsize=32768"


Reply to this email directly or view it on GitHub:
#6658 (comment)

@theuni

This comment has been minimized.

Copy link
Member

@theuni theuni commented Sep 11, 2015

I'd be concerned if no compiler could build on that VPS. However, I'd be willing to bet that it's just a matter of an inefficient gcc version. If clang dies the same way, I'll buy you a beer this weekend :)

@luke-jr

This comment has been minimized.

Copy link
Member

@luke-jr luke-jr commented Sep 11, 2015

@TheBlueMatt Memory use is not an issue, if it correctly builds with the appropriate CXXFLAGS... IMO

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Sep 11, 2015

This is a default gcc on a relatively recent Debian. I really don't think the response to "I can't build Bitcoin Core on a vps with 1GB of memory with default parameters" should be "oh, but if you set these flags you can!"

On September 11, 2015 12:34:57 AM EDT, Cory Fields notifications@github.com wrote:

I'd be concerned if no compiler could build on that VPS. However, I'd
be willing to bet that it's just a matter of an inefficient gcc
version. If clang dies the same way, I'll buy you a beer this weekend
:)


Reply to this email directly or view it on GitHub:
#6658 (comment)

@unsystemizer

This comment has been minimized.

Copy link
Contributor

@unsystemizer unsystemizer commented Sep 11, 2015

0.11 does require more RAM and in my experience --disable-tests doesn't help.
0.10 could be built on systems with 512MB of RAM (with and without --disable-tests).

@jgarzik

This comment has been minimized.

Copy link
Contributor

@jgarzik jgarzik commented Sep 11, 2015

Any instrumentation on why script_test takes so much memory? Would prefer to discern "why" before taking any action.

In general we should be able to build Bitcoin Core on a 1GB VPS.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented Sep 11, 2015

Y'all being overly pedantic. Matt identified a usability problem: User obtains and attempts to compile the software with normal instructions on a common and reasonable platform type, and it crashes.

If you want to suggest that the docs should have instructions with low mem cflags or whatnot, OKAY, but without that it doesn't stop being an issue because just us experts can work around it. Our memory usage is kind of obnoxious during compile time IMO, and if we could do some minor tweaks to reduce the peak that would be nice too-- but at the very least we should provide some instructions.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 18, 2015

This is actually known and documented: https://github.com/bitcoin/bitcoin/blob/master/doc/build-unix.md#system-requirements (feel free to add the recommendation for low-memory build flags tho!)

I would have expected that no longer using boost::spirit (for JSON) and boost::asio (for RPC) would reduce compilation memory usage - unfortunate to hear that it became worse.

Anyhow: cross compilation and pseudo-static compilation is extremely easy with the depends system, I'd recommend using that.

@laanwj laanwj closed this Sep 18, 2015
@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 18, 2015

Oh wait, the compilation of that file uses more than 1GB of memory?
Ok, reopening, that is definitly not good.

@laanwj laanwj reopened this Sep 18, 2015
@laanwj laanwj added the Build system label Sep 18, 2015
@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Sep 18, 2015

Yes, the VPS in question has a total of 1.5GB memory (1GB ram + .5 swap), and nothing else of note was running at the time.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 29, 2015

I measured using gcc 4.8.4 (64 bit): Top 10 memory usage during build:

nr source elapsed maxresident (kB)
1. main.cpp 0:36.58 1248524
2. init.cpp 0:24.24 964136
3. wallet/wallet.cpp 0:22.11 896096
4. test/script_tests.cpp 0:19.44 859708
5. net.cpp 0:18.85 813836
6. wallet/rpcwallet.cpp 0:14.16 727956
7. wallet/walletdb.cpp 0:13.48 719388
8. wallet/rpcdump.cpp 0:12.42 697660
9. qt/walletmodel.cpp 0:11.26 680500
10. rpcrawtransaction.cpp 0:11.44 666548

Will try with clang some other time.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 30, 2015

Same information, but for clang 3.6.0 (also 64-bit):

nr source elapsed maxresident
1 main.cpp 0:28.34 723960
2 wallet/wallet.cpp 0:15.93 522068
3 init.cpp 0:19.06 517080
4 net.cpp 0:14.36 428924
5 wallet/rpcwallet.cpp 0:11.53 357224
6 wallet/walletdb.cpp 0:10.16 356792
7 test/script_tests.cpp 0:12.73 342376
8 qt/walletmodel.cpp 0:08.79 333660
9 test/test_bitcoin.cpp 0:08.21 326100
10 wallet/rpcdump.cpp 0:09.19 322376

The contenders are the (mostly) same, although it is faster and uses significantly less memory in all cases. So we should add a recommendation to use clang on lower-memory systems.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 30, 2015

I'm not sure why it is this "bad". main and init have huge complex implementation files, but of script_tests you'd say it isn't that bad - just 1016 lines.

Possibly it has to do with embedding the test data as includes:

410344  test/data/script_invalid.json.h
530534  test/data/script_valid.json.h
@theuni

This comment has been minimized.

Copy link
Member

@theuni theuni commented Sep 30, 2015

@laanwj I suspect you're right about the test data. Maybe older gcc re-reads that each time it's needed and balloons in size as a result.

@TheBlueMatt Could you test with theuni@1a107da and see if that helps? That moves the test data into cpps instead.
Edit: Whoops. Here's one that actually works: theuni@4b69de1

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Oct 1, 2015

Hmm, the VPS has since been upgraded to have more memory (yay pruning means I get more memory and less disk space...) and doesnt kill script_tests anymore :( What may have been happening is the host may have been retroactively killing processes after some other process had casued the VM to hit some high memory bar. In any case, it is just Debian Jessie, so gcc 4.9.2-10.

@TheBlueMatt

This comment has been minimized.

Copy link
Contributor Author

@TheBlueMatt TheBlueMatt commented Oct 1, 2015

Still, main max resident of 1.2GB will probably not build on most VPSes today, but sadly thats a harder one to fix, and probably a very long-term thing.

@gmaxwell

This comment has been minimized.

Copy link
Contributor

@gmaxwell gmaxwell commented Oct 1, 2015

Rather than recommending clang (which would send more hosts into a less tested and potentially consensus inconsistent config) we could recommend compiler flags that reduce memory usage.

@unsystemizer

This comment has been minimized.

Copy link
Contributor

@unsystemizer unsystemizer commented Oct 1, 2015

I ran a comparison test yesterday measuring the peak memory consumption of the Bash shell session in which I was working. gcc - 835 MB, clang - 823 MB (32-bit Raspbian OS). Not a big saving. (I don't know how @laanwj measured RAM consumption, otherwise I would have used his approach).

People low on RAM could use binary builds if they don't have enough RAM to build, or build without tests. Not everyone should be able to build from source on their system (or, in most hosting environments, they can upsize their RAM, build, and then downsize?).

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Oct 1, 2015

@gmaxwell Possibly both - we're using clang for OSX builds already, so it's not that much of an unknown. I've switched to clang for my run-of-the-mill builds on Linux to save time.

@cfields The solution in https://stackoverflow.com/questions/5479691/is-there-any-standard-way-of-embedding-resources-into-linux-executable-image to use the .incbin directive is also quite nice - create a file blob.S:

    .global blob
    .global blob_size
    .section .rodata
blob:
    .incbin "blob.bin"
1:
blob_size:
    .int 1b - blob

which can just be compiled with gcc along with the rest of the source files. This avoids any kind of conversion.

@unsystemizer see https://gist.github.com/laanwj/108877a28ec03836568a

In general I recommend to use cross-compiling to build for resource-constrained systems. This a) avoids having a compiler toolchain on the target b) is much faster c) Bitcoin Core's depends system makes this easy.

@jarret

This comment has been minimized.

Copy link
Contributor

@jarret jarret commented Jan 25, 2016

After tripping up on this initially, I succeeded in applying Luke-jr's
workaround by giving the CXXFLAGS at configure time:

$ ./configure --disable-wallet CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768"

On a Raspberry Pi 2 building with a single core:

$ time make
(success)
real    159m41.895s
user    154m55.090s
sys     3m1.710s

The memory use seemed to stay below 500 MB throughout, but I didn't measure
that precisely.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Jun 5, 2018

Sorry to say, but 1GB of RAM and 4 cores is not useful for compilation. In my experience reserving memory for VMs, 1-1.5GB per -jX parallelism works, more is pushing it, it can be done with lots of swap space but you won't win time! For ARM I (still) strongly recommend to use cross-compilation.

@Sjors

This comment has been minimized.

Copy link
Member

@Sjors Sjors commented Jun 5, 2018

I'm looking at Armian builds, since I might as well just build the rest of the OS - and sync the chain - along with cross compiling bitcoind :-)

@bitcoin bitcoin deleted a comment from Rosswestwood Jun 11, 2018
@MarcoFalke

This comment has been minimized.

Copy link
Member

@MarcoFalke MarcoFalke commented Jun 21, 2018

Considering that clang uses a lot less memory compared to gcc, why don't we mention clang as an alternative in https://github.com/bitcoin/bitcoin/blob/000abbb6b07410357a928768d7d56465ba0d3bac/doc/build-unix.md#memory-requirements

Should I create a pull for this?

@dufferzafar

This comment has been minimized.

Copy link

@dufferzafar dufferzafar commented Sep 1, 2018

@MarcoFalke:

why don't we mention clang as an alternative

@gmaxwell said above:

Rather than recommending clang (which would send more hosts into a less tested and potentially consensus inconsistent config) we could recommend compiler flags that reduce memory usage.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 1, 2018

well if clang, a compiler many, many people use, generates consensus incompatible code, bitcoin has a huge problem.

(to be clear, no, I don't think this is an issue worrying about here - let's focus on finding configurations that reduce memory usage)

@dufferzafar

This comment has been minimized.

Copy link

@dufferzafar dufferzafar commented Sep 1, 2018

To be honest, I myself use clang (along with ccache) but that's only because all my work involves running in regtest, so I don't have to worry about consensus compatibility.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 1, 2018

I mean, it might have been a good argument in 2012, but at this point, many platforms are clang-only (MacOSX, FreeBSD, OpenBSD, at least). There's nothing that binds bitcoin to gcc specifically.

@meeDamian

This comment has been minimized.

Copy link
Contributor

@meeDamian meeDamian commented Sep 11, 2018

Just wanted to leave an update here, that building Bitcoin Core v0.16.2 is possible¹ on Raspberry Pi Zero W, which has 512MB RAM + default of 100MB SWAP.

./autogen.sh
./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" --enable-cxx --without-gui --disable-shared --with-pic
--enable-upnp-default --disable-wallet --disable-tests
make
sudo make install

¹ It did take a very long time.

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Sep 11, 2018

@meeDamian hahaha nice, that's quite a thing, building with less than a GB RAM+SWAP

@practicalswift

This comment has been minimized.

Copy link
Member

@practicalswift practicalswift commented Jun 27, 2019

As noted in this issue we are currently using significantly more compile-time memory than we technically have to.

I plan to address this and you can see steps being taken in #16273 and #16278 (the latter is a subset of the former limited to tests only).

People who care about compile-time memory usage might be interested in reviewing those PR:s :-)

@practicalswift

This comment has been minimized.

Copy link
Member

@practicalswift practicalswift commented Jun 27, 2019

These are the 25 worst offenders in terms of compile-time memory usage:

# File Max memory usage (RSS)
1 test/test_test_bitcoin-script_tests.o 1416 MB
2 wallet/libbitcoin_wallet_a-wallet.o 1102 MB
3 wallet/libbitcoin_wallet_a-rpcwallet.o 1059 MB
4 libbitcoin_server_a-validation.o 953 MB
5 test/test_test_bitcoin-util_tests.o 905 MB
6 wallet/test/test_test_bitcoin-wallet_tests.o 905 MB
7 wallet/test/test_test_bitcoin-coinselector_tests.o 897 MB
8 libbitcoin_server_a-init.o 892 MB
9 qt/test/qt_test_test_bitcoin_qt-wallettests.o 890 MB
10 wallet/test/test_test_bitcoin-ismine_tests.o 874 MB
11 qt/test/qt_test_test_bitcoin_qt-addressbooktests.o 863 MB
12 libbitcoin_server_a-net_processing.o 857 MB
13 wallet/test/test_test_bitcoin-psbt_wallet_tests.o 837 MB
14 test/test_test_bitcoin-transaction_tests.o 813 MB
15 qt/qt_libbitcoinqt_a-sendcoinsdialog.o 809 MB
16 wallet/libbitcoin_wallet_a-rpcdump.o 800 MB
17 test/test_test_bitcoin-rpc_tests.o 792 MB
18 libbitcoin_server_a-validationinterface.o 770 MB
19 test/test_bitcoin 752 MB
20 rpc/libbitcoin_server_a-blockchain.o 745 MB
21 qt/qt_libbitcoinqt_a-rpcconsole.o 741 MB
22 test/test_test_bitcoin-sighash_tests.o 739 MB
23 rpc/libbitcoin_server_a-rawtransaction.o 721 MB
24 qt/qt_libbitcoinqt_a-walletmodel.o 721 MB
25 wallet/libbitcoin_wallet_a-walletdb.o 720 MB
... ... ...
449 support/libbitcoin_util_a-cleanse.o 24 MB
450 crypto/sha256_sse4.cpp 24 MB
451 compat/libbitcoin_util_a-glibc_sanity.o 24 MB
452 script/script_error.cpp 23 MB
453 script/libbitcoin_consensus_a-script_error.o 23 MB
454 compat/libbitcoin_util_a-strnlen.o 18 MB

Measured using gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0.

@practicalswift

This comment has been minimized.

Copy link
Member

@practicalswift practicalswift commented Jun 29, 2019

People interested in enabling builds on low memory machines might be interested in reviewing and testing #16312:

  • tests: Reduce compile-time memory usage by 0.5 GB when compiling script_tests.cpp (from 1.4 GB to 0.9 GB)
@practicalswift

This comment has been minimized.

Copy link
Member

@practicalswift practicalswift commented Aug 10, 2019

Chasing {concept,approach} ACK/NACKs for #16331 which tackles this issue.

Let me know if this issue is no longer relevant :-)

@laanwj

This comment has been minimized.

Copy link
Member

@laanwj laanwj commented Aug 12, 2019

@meeDamian

This comment has been minimized.

Copy link
Contributor

@meeDamian meeDamian commented Aug 12, 2019

I'm not sure if I'm not missing something here, but bitcoind was already building on RBP zero (512MB ram with just 100MB swap). Plus, I've just built v0.18.1 on RBP3 (1GB ram, no swap) a few days ago without problems even before applying any PRs. It does take its time, however 😅. So I for once, am not sure what else/how to test?

@Sjors

This comment has been minimized.

Copy link
Member

@Sjors Sjors commented Aug 12, 2019

@meeDamian does the new PR work without those extra config flags?

@MarcoFalke

This comment has been minimized.

Copy link
Member

@MarcoFalke MarcoFalke commented Aug 12, 2019

It is a bit concerning that the unit test have to be disabled to compile on low memory machines

@meeDamian

This comment has been minimized.

Copy link
Contributor

@meeDamian meeDamian commented Aug 13, 2019

On a fresh Raspbian on RBP3, running (with or without --enable-lowmem flag):

sudo swapoff --all

sudo apt install git build-essential libtool autotools-dev automake pkg-config bsdmainutils python3 libssl-dev libevent-dev libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev libminiupnpc-dev libzmq3-dev

git clone  -b fix-absurd-memory-usage-when-compiling-script_build  https://github.com/practicalswift/bitcoin.git bitcoin-swift
cd bitcoin-swift/

./contrib/install_db4.sh $(pwd)
export BDB_PREFIX=$(pwd)/db4
./autogen.sh

./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" --enable-lowmem --without-gui

Crashes with:

…
checking for upnpDiscover in -lminiupnpc... yes
checking miniupnpc/miniupnpc.h usability... yes
checking miniupnpc/miniupnpc.h presence... yes
checking for miniupnpc/miniupnpc.h... yes
checking for upnpDiscover in -lminiupnpc... (cached) yes
checking miniupnpc/upnpcommands.h usability... yes
checking miniupnpc/upnpcommands.h presence... yes
checking for miniupnpc/upnpcommands.h... yes
checking for upnpDiscover in -lminiupnpc... (cached) yes
checking miniupnpc/upnperrors.h usability... yes
checking miniupnpc/upnperrors.h presence... yes
checking for miniupnpc/upnperrors.h... yes
checking for upnpDiscover in -lminiupnpc... (cached) yes
checking whether to build Bitcoin Core GUI... no (Qt5)
checking for boostlib >= 1.47.0 (104700)... yes
checking whether the Boost::System library is available... yes
configure: error: Could not find a version of the library!

Running the same on bitcoin v0.18.1 does not crash.

@meeDamian

This comment has been minimized.

Copy link
Contributor

@meeDamian meeDamian commented Aug 13, 2019

Actually… bitcoin master also crashes the same way, so likely unrelated to this. Happy to try again if someone advises how to fix/proceed.

@Sjors

This comment has been minimized.

Copy link
Member

@Sjors Sjors commented Aug 13, 2019

@meeDamian did you try with practicalswift/low-mem? #16331

@meeDamian

This comment has been minimized.

Copy link
Contributor

@meeDamian meeDamian commented Aug 13, 2019

I'm afraid practicalswift/low-mem crashes the same way as master, and fix-absurd-memory-usage-when-compiling-script_build. Any idea what library version might be missing there, or how to fix it?

fanquake added a commit that referenced this issue Aug 14, 2019
fa7789f doc: Mention other ways to conserve memory on compilation (MarcoFalke)

Pull request description:

  Related to:

  *  Building requires >1GB memory #6658

ACKs for top commit:
  instagibbs:
    ACK fa7789f
  kristapsk:
    ACK fa7789f

Tree-SHA512: 828593de9cfa9f9027c6c8e97abe95e3fad9f2ff50e6512808a8950de4a1f9ea901e724edfb96d7119224c3e38a136e60cb798b34ef682c585a7951e02124a3a
@MarcoFalke

This comment has been minimized.

Copy link
Member

@MarcoFalke MarcoFalke commented Oct 12, 2019

The docs are updated and mention that clang can be used, which uses less than 1 GB of memory. Closing for now

https://github.com/bitcoin/bitcoin/blob/561a7d30478b82f5d46dcf0f16e864a9608004f4/doc/build-unix.md#memory-requirements

@MarcoFalke MarcoFalke closed this Oct 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.