Program consumes all memory on DB of size comparable to RAM and blows up on Bitcoin blockchain transaction index

[cirosantilli]

LevelDB Dumper version

e750a27

To Reproduce

To obtain the test data, you have to install Bitcoin and fetch a full node with txindex, which can take 24 hours, on Ubuntu 23.10:

snap install bitcoin-core
bitcoin-core.daemon -txindex=1

Also the exact data layout is not deterministic, but hopefully that won't matter 🤞

After that the leveldb is located at /snap/bitcoin-core/common/.bitcoin/indexes/txindex and running:

LevelDBDumper -q -t json -d ~/snap/bitcoin-core/common/.bitcoin/indexes/txindex -o . -f btc.json

consumes more and more memory until after about 30 minutes it takes up everything and the kernel kills it.

The size of all files in the leveldb database directory is ~ 46.2 GiB, so it does not feel very reasonable that it should blow up the 64 GB RAM of my Lenovo ThinkPad P14s. Maybe there is some memory that could be freed but isn't?

The question is, is it possible to read the input files bit by bit, and output as we go, or does it absolutely require reading everything into memory.