Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bitcoin-core multiwallets limited to about 300 loads #15291

Closed
fscemama opened this issue Jan 30, 2019 · 28 comments · Fixed by #15297
Closed

Bitcoin-core multiwallets limited to about 300 loads #15291

fscemama opened this issue Jan 30, 2019 · 28 comments · Fixed by #15297

Comments

@fscemama
Copy link

@fscemama fscemama commented Jan 30, 2019

Hi,
Using Bitcoin Core RPC client version v0.17.0.1-g1476554d3b1db5352c0de1de7524864d4bce3aac

I'm using bitcoin-cli loadwallet and unloadwallet commands. They both seem to work fine.
Unfortunately, memory is getting higher and higher, and no wallet can be loaded after 300 have been.
The problem may be related to BerkeleyDB rather than to Bitcoin Core itself. Fact is that lsof shows handles not cleared off after the unloadwallet command is issued. Files still handled are .walletlock, db.log and database/log.0000000001 .
When 300 different wallets have been loaded/unloaded, used RAM is over 8 GB, and bitcoind will finally become unstable and fail.

Any help would be appreciated.
Regards,

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 30, 2019

Are you doing some stress test or are these load/unload done over a long period of time?

Is the same wallet loaded/unloaded multiple times?

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 30, 2019

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 30, 2019

unloadwallet behavior was changed in #14941, you could try that (it was also backport to 0.17 branch in #15002).

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 30, 2019

Thank you for the reference. I'm going to give it a test. Agreed, an asynchrous unloadwallet can be useful (faster), but can also lead to disasters. I've always been surprised that BerkeleyDB 4 should be used, but that's another story.

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 30, 2019

@fscemama thanks, please let us know how it went.

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 30, 2019

Hi, just stress-tested with today's github master. Result is alas same.
Basic testing:
Firstly, load 30 wallets:
j=0; for i in /wallets/*; do j=echo $j+1|bc; if [ $j -lt 30 ]; then bitcoin-cli loadwallet "$i"; fi; done
Then, unload them:
for i in /wallets/*;do bitcoin-cli unloadwallet "$i"; done
Look for handles:
lsof -p pidof bitcoind |grep wallet
89 lines, .walletlock, db.log, etc. As for memory, 1.5 GB are used by bitcoind.
I must mention unloadwallet commands have taken no time to execute.
Thanks !

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

But do you see memory leak?

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

I'm not sure about what is a memory leak. Surely, the memory footprint remains very high (1.5 GB for 30 loaded wallets, and no memory is released after wallets are unloaded). In fact, lsof return proves wallets db files still being open. Unloadwallet does not close DB files.
BTW, loading one more wallet should not take 400+ MB of memory, there's certainly something to optimize there, but that's a drawback we can live with.
I'm sorry, I have no C++ knowledge to give any useful help in code.

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

If each wallet takes 400+ MB then with 30 wallets you would need much more memory than 1.5GB. Can you check memory usage after each loadwallet and then after each unloadwallet. You could test #15297 branch which closes those files?

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

40-50 MB. Sorry, end of week!!

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

I agree to test, please let me know when I can download the master. Then I'll compile and let you know. The hack proposed in #15297 looks fine.

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

It would help if you test that branch instead of master:

git fetch git@github.com:bitcoin/bitcoin.git refs/pull/15297/head
git checkout -b pr15297 FETCH_HEAD
make -j
...
@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

THANKS.

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

Still unrelased handles. Sample output of the lsof command:

bitcoind 1568 root   37uW     REG      8,1         0   670564 /wallets/xxxx/.walletlock
bitcoind 1568 root   38w      REG      8,1         0   670567 /wallets/xxxx/db.log
bitcoind 1568 root   41u      REG      8,1   1048576   654805 /wallets/xxxx/database/log.0000000001

I've considered the possibility of clearing off these file handles the kernel way, like this:
echo -e "p close(654805)\nquit\ny\n" | gdp -p 1568
And this is not a good idea : BerkeleyDB will complain and this could lead to database corruption.
Maybe I've missed something, the #15297 thread makes me feel like trying to google the BDB issue and try to patch it.

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

I'm a newbie as for C++ and I may be wrong, but I'm not sure the closing of the WalletDatabase is ever called.

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

Mind sharing the steps to reproduce here?

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

No more than that:

Basic testing:
Firstly, load 30 wallets:
j=0; for i in /wallets/*; do j=echo $j+1|bc; if [ $j -lt 30 ]; then bitcoin-cli loadwallet "$i"; fi; done
Then, unload them:
for i in /wallets/*;do bitcoin-cli unloadwallet "$i"; done
Look for handles:
lsof -p pidof bitcoind |grep wallet
89 lines, .walletlock, db.log, etc. As for memory, 1.5 GB are used by bitcoind.

Regards,

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

@fscemama are you sure you built #15297 branch? I can't reproduce your case.

Anyway, I'll push a commit later testing that the files should not exists after unloadwallet calls.

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

Really got it. Bitcoin Core RPC client version v0.17.0.1-g1476554d3b1db5352c0de1de7524864d4bce3aac

My part of db.cpp:

void BerkeleyEnvironment::Close()
{
    if (!fDbEnvInit)
        return;

    fDbEnvInit = false;

    for (auto& db : m_databases) {
        auto count = mapFileUseCount.find(db.first);
        assert(count == mapFileUseCount.end() || count->second == 0);
        BerkeleyDatabase& database = db.second.get();
        if (database.m_db) {
            database.m_db->close(0);
            database.m_db.reset();
        }
    }

    FILE* error_file = nullptr;
    dbenv->get_errfile(&error_file);

    int ret = dbenv->close(0);
    if (ret != 0)
        LogPrintf("BerkeleyEnvironment::Close: Error %d closing database environment: %s\n", ret, DbEnv::strerror(ret));
    if (!fMockDb)
        DbEnv((u_int32_t)0).remove(strPath.c_str(), 0);

    if (error_file) fclose(error_file);

    UnlockDirectory(strPath, ".walletlock");
}

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

Sorry, what gives ./src/bitcoind --version?

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

For some reason, make install in master-bitcoin dit not overwrite /usr/local/bin/bitcoind. I've done it manually and retested. I confirm the file handles are cleared off. Memory footprint is not reduced, though.

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

# /usr/local/bin/bitcoind --version
Bitcoin Core Daemon version v0.17.99.0-bdbb3b5a8
Copyright (C) 2009-2019 The Bitcoin Core developers

Please contribute if you find Bitcoin Core useful. Visit
<https://bitcoincore.org> for further information about the software.
The source code is available from <https://github.com/bitcoin/bitcoin>.

This is experimental software.
Distributed under the MIT software license, see the accompanying file COPYING
or <https://opensource.org/licenses/MIT>

This product includes software developed by the OpenSSL Project for use in the
OpenSSL Toolkit <https://www.openssl.org> and cryptographic software written by
Eric Young and UPnP software written by Thomas Bernard.

@sipa

This comment has been minimized.

Copy link
Member

@sipa sipa commented Jan 31, 2019

Releasing memory back to the system is often nontrivial due to fragmentation.

The bigger question is whether if you first load 300 wallets, then unload them, and then load 300 (other) wallets, does the memory usage stay constant the second time, or does it go up further?

@promag

This comment has been minimized.

Copy link
Member

@promag promag commented Jan 31, 2019

Thanks for testing @fscemama and confirming the issue is fixed.

Regarding memory, @sipa is right, a good test should use lots of non empty wallets (with lots of transactions and keys).

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

Thanks @promag and @sipa for making Bitcoin Core better.
Here are a few tests, which may help understand why memory is not released.

bitcoin-cli stop
...restart...
ls -l /root/wallets|wc
#     160    1433   12484
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 10 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 988168
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 988696
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 10 ] && [ $j -lt 20 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 990.3m
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 990.3m
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 20 ] && [ $j -lt 30 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 0.991g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 0.991g
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 30 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 1.415g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 1.415g
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 30 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 1.439g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 1.439g
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 30 ] && [ $j -lt 60 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 1.462g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 1.461g
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 60 ] && [ $j -lt 90 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 1.485g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 1.485g
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 90 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 2.582g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 2.582g
j=0; for i in /root/wallets/*; do j=`echo $j+1|bc`; if [ $j -gt 0 ] && [ $j -lt 90 ]; then echo $j; bitcoin-cli loadwallet $i; fi; done
# memory: 2.606g
for i in `bitcoin-cli listwallets|grep wallets|sed "s|\,$||"|sed "s|\"||g"`; do echo $i; bitcoin-cli unloadwallet $i; done
# memory: 2.607g

lsof -p `pidof bitcoind`

COMMAND   PID USER   FD      TYPE             DEVICE  SIZE/OFF      NODE NAME
bitcoind 2862 root  cwd       DIR                8,3      4096  99876868 /home/bitcoin/testnet3
bitcoind 2862 root  rtd       DIR                8,2      4096         2 /
bitcoind 2862 root  txt       REG                8,2 122945248    156543 /usr/local/bin/bitcoind
bitcoind 2862 root  mem       REG                8,3   2180920  99877103 /home/bitcoin/testnet3/chainstate/018395.ldb
bitcoind 2862 root  mem       REG                8,3   2179799  99891374 /home/bitcoin/testnet3/chainstate/009873.ldb
bitcoind 2862 root  mem       REG                8,3   2174556  99877377 /home/bitcoin/testnet3/chainstate/019086.ldb
bitcoind 2862 root  mem       REG                8,3   2173092  99877375 /home/bitcoin/testnet3/chainstate/019084.ldb
bitcoind 2862 root  mem       REG                8,3   2176542  99877374 /home/bitcoin/testnet3/chainstate/019083.ldb
bitcoind 2862 root  mem       REG                8,2     84848    917110 /lib/x86_64-linux-gnu/libresolv-2.24.so
bitcoind 2862 root  mem       REG                8,2     47632    917022 /lib/x86_64-linux-gnu/libnss_files-2.24.so
bitcoind 2862 root  mem       REG                8,3   2180040  99891355 /home/bitcoin/testnet3/chainstate/010262.ldb
bitcoind 2862 root  mem       REG                8,3   2182434  99876958 /home/bitcoin/testnet3/chainstate/018475.ldb
bitcoind 2862 root  mem       REG                8,3   2175576  99877027 /home/bitcoin/testnet3/chainstate/018391.ldb
bitcoind 2862 root  mem       REG                8,3   2179082  99877025 /home/bitcoin/testnet3/chainstate/018380.ldb
bitcoind 2862 root  mem       REG                8,3   2174953  99876890 /home/bitcoin/testnet3/chainstate/018342.ldb
bitcoind 2862 root  mem       REG                8,3   2175645  99876892 /home/bitcoin/testnet3/chainstate/018480.ldb
bitcoind 2862 root  mem       REG                8,3   2183696  99877147 /home/bitcoin/testnet3/chainstate/018412.ldb
bitcoind 2862 root  mem       REG                8,3   2177506  99876959 /home/bitcoin/testnet3/chainstate/018345.ldb
bitcoind 2862 root  mem       REG                8,3   2176305  99877277 /home/bitcoin/testnet3/chainstate/018618.ldb
bitcoind 2862 root  mem       REG                8,3   2177619  99876984 /home/bitcoin/testnet3/chainstate/018419.ldb
bitcoind 2862 root  mem       REG                8,3   2182692  99891431 /home/bitcoin/testnet3/chainstate/003178.ldb
bitcoind 2862 root  mem       REG                8,3   2179768  99877121 /home/bitcoin/testnet3/chainstate/018359.ldb
bitcoind 2862 root  mem       REG                8,3   2181562  99877243 /home/bitcoin/testnet3/chainstate/018612.ldb
bitcoind 2862 root  mem       REG                8,3   2178768  99891433 /home/bitcoin/testnet3/chainstate/003179.ldb
bitcoind 2862 root  mem       REG                8,3   2182919  99891799 /home/bitcoin/testnet3/chainstate/003133.ldb
bitcoind 2862 root  mem       REG                8,3   2176229  99876887 /home/bitcoin/testnet3/chainstate/018340.ldb
bitcoind 2862 root  mem       REG                8,3   2180656  99891669 /home/bitcoin/testnet3/chainstate/005762.ldb
bitcoind 2862 root  mem       REG                8,3   2177080  99891451 /home/bitcoin/testnet3/chainstate/009564.ldb
bitcoind 2862 root  mem       REG                8,3   2181179  99890591 /home/bitcoin/testnet3/chainstate/003108.ldb
bitcoind 2862 root  mem       REG                8,3   2181264  99877029 /home/bitcoin/testnet3/chainstate/018382.ldb
bitcoind 2862 root  mem       REG                8,3   2177242  99890345 /home/bitcoin/testnet3/chainstate/008057.ldb
bitcoind 2862 root  mem       REG                8,3   2181907  99890437 /home/bitcoin/testnet3/chainstate/009699.ldb
bitcoind 2862 root  mem       REG                8,3   2178513  99890553 /home/bitcoin/testnet3/chainstate/009866.ldb
bitcoind 2862 root  mem       REG                8,3   2177449  99891315 /home/bitcoin/testnet3/chainstate/006630.ldb
bitcoind 2862 root  mem       REG                8,3   2177974  99892097 /home/bitcoin/testnet3/chainstate/010241.ldb
bitcoind 2862 root  mem       REG                8,3   2177291  99891201 /home/bitcoin/testnet3/chainstate/009639.ldb
bitcoind 2862 root  mem       REG                8,3   2177460  99877233 /home/bitcoin/testnet3/chainstate/018495.ldb
bitcoind 2862 root  mem       REG                8,3   2181081  99891351 /home/bitcoin/testnet3/chainstate/009733.ldb
bitcoind 2862 root  mem       REG                8,3   2177924  99877118 /home/bitcoin/testnet3/chainstate/018398.ldb
bitcoind 2862 root  mem       REG                8,3   2176261  99877033 /home/bitcoin/testnet3/chainstate/018406.ldb
bitcoind 2862 root  mem       REG                8,3   2177667  99876962 /home/bitcoin/testnet3/chainstate/018476.ldb
bitcoind 2862 root  mem       REG                8,3   2178498  99891398 /home/bitcoin/testnet3/chainstate/003300.ldb
bitcoind 2862 root  mem       REG                8,3   2175006  99877010 /home/bitcoin/testnet3/chainstate/018347.ldb
bitcoind 2862 root  mem       REG                8,3   2182108  99891788 /home/bitcoin/testnet3/chainstate/003216.ldb
bitcoind 2862 root  mem       REG                8,3   2176581  99877128 /home/bitcoin/testnet3/chainstate/018364.ldb
bitcoind 2862 root  mem       REG                8,3   2182743  99889425 /home/bitcoin/testnet3/chainstate/003161.ldb
bitcoind 2862 root  mem       REG                8,3   2179578  99891379 /home/bitcoin/testnet3/chainstate/010397.ldb
bitcoind 2862 root  mem       REG                8,3   2182454  99891715 /home/bitcoin/testnet3/chainstate/003211.ldb
bitcoind 2862 root  mem       REG                8,3   2174949  99877122 /home/bitcoin/testnet3/chainstate/018360.ldb
bitcoind 2862 root  mem       REG                8,3   2182177  99891031 /home/bitcoin/testnet3/chainstate/009558.ldb
bitcoind 2862 root  mem       REG                8,3   2177919  99877235 /home/bitcoin/testnet3/chainstate/018497.ldb
bitcoind 2862 root  mem       REG                8,3   2183410  99891519 /home/bitcoin/testnet3/chainstate/003312.ldb
bitcoind 2862 root  mem       REG                8,3   2177699  99891771 /home/bitcoin/testnet3/chainstate/008062.ldb
bitcoind 2862 root  mem       REG                8,3   2182885  99891367 /home/bitcoin/testnet3/chainstate/009871.ldb
bitcoind 2862 root  mem       REG                8,3   2178331  99890491 /home/bitcoin/testnet3/chainstate/003173.ldb
bitcoind 2862 root  mem       REG                8,3   2177929  99891055 /home/bitcoin/testnet3/chainstate/010257.ldb
bitcoind 2862 root  mem       REG                8,3   2182026  99876997 /home/bitcoin/testnet3/chainstate/018427.ldb
bitcoind 2862 root  mem       REG                8,3   2177335  99877213 /home/bitcoin/testnet3/chainstate/018465.ldb
bitcoind 2862 root  mem       REG                8,3   2176444  99892072 /home/bitcoin/testnet3/chainstate/009794.ldb
bitcoind 2862 root  mem       REG                8,3   2176904  99891310 /home/bitcoin/testnet3/chainstate/009641.ldb
bitcoind 2862 root  mem       REG                8,3   2177353  99877223 /home/bitcoin/testnet3/chainstate/018483.ldb
bitcoind 2862 root  mem       REG                8,3   2183308  99876900 /home/bitcoin/testnet3/chainstate/018469.ldb
bitcoind 2862 root  mem       REG                8,3   2178994  99891811 /home/bitcoin/testnet3/chainstate/003219.ldb
bitcoind 2862 root  mem       REG                8,3   2178455  99877144 /home/bitcoin/testnet3/chainstate/018366.ldb
bitcoind 2862 root  mem       REG                8,3   2178159  99891313 /home/bitcoin/testnet3/chainstate/003174.ldb
bitcoind 2862 root  mem       REG                8,3   2174213  99877116 /home/bitcoin/testnet3/chainstate/018357.ldb
bitcoind 2862 root  mem       REG                8,3   2182530  99891407 /home/bitcoin/testnet3/chainstate/009672.ldb
bitcoind 2862 root  mem       REG                8,3   2176661  99876921 /home/bitcoin/testnet3/chainstate/011680.ldb
bitcoind 2862 root  mem       REG                8,3   2180478  99890336 /home/bitcoin/testnet3/chainstate/009886.ldb
bitcoind 2862 root  mem       REG                8,3   2182688  99891741 /home/bitcoin/testnet3/chainstate/009704.ldb
bitcoind 2862 root  mem       REG                8,3   2181569  99880806 /home/bitcoin/testnet3/chainstate/009787.ldb
bitcoind 2862 root  mem       REG                8,3   2179517  99877205 /home/bitcoin/testnet3/chainstate/018461.ldb
bitcoind 2862 root  mem       REG                8,3   2175639  99891738 /home/bitcoin/testnet3/chainstate/010413.ldb
bitcoind 2862 root  mem       REG                8,3   2184423  99890271 /home/bitcoin/testnet3/chainstate/003164.ldb
bitcoind 2862 root  mem       REG                8,3   2181247  99891587 /home/bitcoin/testnet3/chainstate/003112.ldb
bitcoind 2862 root  mem       REG                8,3   2177058  99877078 /home/bitcoin/testnet3/chainstate/018385.ldb
bitcoind 2862 root  mem       REG                8,3   2181950  99890418 /home/bitcoin/testnet3/chainstate/010245.ldb
bitcoind 2862 root  mem       REG                8,3   2180119  99889728 /home/bitcoin/testnet3/chainstate/009732.ldb
bitcoind 2862 root  mem       REG                8,3   2181931  99891810 /home/bitcoin/testnet3/chainstate/003136.ldb
bitcoind 2862 root  mem       REG                8,3   2175033  99876888 /home/bitcoin/testnet3/chainstate/018341.ldb
bitcoind 2862 root  mem       REG                8,3   2183125  99890558 /home/bitcoin/testnet3/chainstate/009780.ldb
bitcoind 2862 root  mem       REG                8,3   2177796  99876875 /home/bitcoin/testnet3/chainstate/011679.ldb
bitcoind 2862 root  mem       REG                8,3   2176473  99891244 /home/bitcoin/testnet3/chainstate/009640.ldb
bitcoind 2862 root  mem       REG                8,3   2181659  99891476 /home/bitcoin/testnet3/chainstate/009692.ldb
bitcoind 2862 root  mem       REG                8,3   2175739  99891854 /home/bitcoin/testnet3/chainstate/008066.ldb
bitcoind 2862 root  mem       REG                8,3   2175583  99877276 /home/bitcoin/testnet3/chainstate/018617.ldb
bitcoind 2862 root  mem       REG                8,3   2183806  99891823 /home/bitcoin/testnet3/chainstate/009770.ldb
bitcoind 2862 root  mem       REG                8,3   2181095  99891518 /home/bitcoin/testnet3/chainstate/009604.ldb
bitcoind 2862 root  mem       REG                8,3   2180066  99891221 /home/bitcoin/testnet3/chainstate/009689.ldb
bitcoind 2862 root  mem       REG                8,3   2180697  99890670 /home/bitcoin/testnet3/chainstate/003111.ldb
bitcoind 2862 root  mem       REG                8,3   2178143  99877034 /home/bitcoin/testnet3/chainstate/018384.ldb
bitcoind 2862 root  mem       REG                8,3   2184015  99891294 /home/bitcoin/testnet3/chainstate/003231.ldb
bitcoind 2862 root  mem       REG                8,3   2179576  99891525 /home/bitcoin/testnet3/chainstate/010407.ldb
bitcoind 2862 root  mem       REG                8,3   2180078  99891428 /home/bitcoin/testnet3/chainstate/010238.ldb
bitcoind 2862 root  mem       REG                8,3   2176392  99891794 /home/bitcoin/testnet3/chainstate/003132.ldb
bitcoind 2862 root  mem       REG                8,3   2174950  99876884 /home/bitcoin/testnet3/chainstate/018339.ldb
bitcoind 2862 root  mem       REG                8,3   2181444  99891806 /home/bitcoin/testnet3/chainstate/003218.ldb
bitcoind 2862 root  mem       REG                8,3   2178471  99877134 /home/bitcoin/testnet3/chainstate/018365.ldb
bitcoind 2862 root  mem       REG                8,3   2181278  99877191 /home/bitcoin/testnet3/chainstate/018453.ldb
bitcoind 2862 root  mem       REG                8,3   2177401  99877237 /home/bitcoin/testnet3/chainstate/018499.ldb
bitcoind 2862 root  mem       REG                8,3   2178140  99891785 /home/bitcoin/testnet3/chainstate/003129.ldb
bitcoind 2862 root  mem       REG                8,3   2173424  99877107 /home/bitcoin/testnet3/chainstate/018387.ldb
bitcoind 2862 root  mem       REG                8,3   2176727  99891495 /home/bitcoin/testnet3/chainstate/010406.ldb
bitcoind 2862 root  mem       REG                8,3   2180570  99890709 /home/bitcoin/testnet3/chainstate/009669.ldb
bitcoind 2862 root  mem       REG                8,3   2182058  99891385 /home/bitcoin/testnet3/chainstate/010398.ldb
bitcoind 2862 root  mem       REG                8,3   2181779  99890529 /home/bitcoin/testnet3/chainstate/003106.ldb
bitcoind 2862 root  mem       REG                8,3   2177996  99877026 /home/bitcoin/testnet3/chainstate/018381.ldb
bitcoind 2862 root  mem       REG                8,3   2179807  99877085 /home/bitcoin/testnet3/chainstate/018394.ldb
bitcoind 2862 root  mem       REG                8,3   2180720  99891824 /home/bitcoin/testnet3/chainstate/009789.ldb
bitcoind 2862 root  mem       REG                8,3   2179210  99891363 /home/bitcoin/testnet3/chainstate/009529.ldb
bitcoind 2862 root  mem       REG                8,3   2177479  99891136 /home/bitcoin/testnet3/chainstate/009637.ldb
bitcoind 2862 root  mem       REG                8,3   2177184  99877182 /home/bitcoin/testnet3/chainstate/018450.ldb
bitcoind 2862 root  mem       REG                8,3   2177910  99891323 /home/bitcoin/testnet3/chainstate/009642.ldb
bitcoind 2862 root  mem       REG                8,3   2181534  99892068 /home/bitcoin/testnet3/chainstate/009791.ldb
bitcoind 2862 root  mem       REG                8,3   2176418  99891442 /home/bitcoin/testnet3/chainstate/009535.ldb
bitcoind 2862 root  mem       REG                8,3   2182250  99891317 /home/bitcoin/testnet3/chainstate/009766.ldb
bitcoind 2862 root  mem       REG                8,3   2177254  99891071 /home/bitcoin/testnet3/chainstate/009576.ldb
bitcoind 2862 root  mem       REG                8,3   2178975  99891289 /home/bitcoin/testnet3/chainstate/009527.ldb
bitcoind 2862 root  mem       REG                8,3   2180133  99891395 /home/bitcoin/testnet3/chainstate/009532.ldb
bitcoind 2862 root  mem       REG                8,3   2178033  99891393 /home/bitcoin/testnet3/chainstate/003297.ldb
bitcoind 2862 root  mem       REG                8,3   2173533  99877009 /home/bitcoin/testnet3/chainstate/018346.ldb
bitcoind 2862 root  mem       REG                8,3   2180343  99890995 /home/bitcoin/testnet3/chainstate/009781.ldb
bitcoind 2862 root  mem       REG                8,3   2180057  99891444 /home/bitcoin/testnet3/chainstate/009617.ldb
bitcoind 2862 root  mem       REG                8,3   2180022  99890470 /home/bitcoin/testnet3/chainstate/009634.ldb
bitcoind 2862 root  mem       REG                8,3   2179722  99891520 /home/bitcoin/testnet3/chainstate/003313.ldb
bitcoind 2862 root  mem       REG                8,3   1578408  99877344 /home/bitcoin/testnet3/blocks/index/021504.ldb
bitcoind 2862 root  mem       REG                8,3   2176121  99889726 /home/bitcoin/testnet3/chainstate/010404.ldb
bitcoind 2862 root  mem       REG                8,3   1120135  99877335 /home/bitcoin/testnet3/blocks/index/021503.ldb
bitcoind 2862 root  mem       REG                8,3   2135889  99877324 /home/bitcoin/testnet3/blocks/index/021502.ldb
bitcoind 2862 root  mem       REG                8,3   2135851  99877322 /home/bitcoin/testnet3/blocks/index/021501.ldb
bitcoind 2862 root  mem       REG                8,3   2136050  99877321 /home/bitcoin/testnet3/blocks/index/021500.ldb
bitcoind 2862 root  mem       REG                8,3   2136114  99877316 /home/bitcoin/testnet3/blocks/index/021499.ldb
bitcoind 2862 root  mem       REG                8,3   2136041  99877313 /home/bitcoin/testnet3/blocks/index/021498.ldb
bitcoind 2862 root  mem       REG                8,3   2135537  99876931 /home/bitcoin/testnet3/blocks/index/021497.ldb
bitcoind 2862 root  mem       REG                8,3   2137271  99890379 /home/bitcoin/testnet3/blocks/index/006674.ldb
bitcoind 2862 root  mem       REG                8,3   2139535  99890378 /home/bitcoin/testnet3/blocks/index/006673.ldb
bitcoind 2862 root  mem       REG                8,3   2137524  99890377 /home/bitcoin/testnet3/blocks/index/006672.ldb
bitcoind 2862 root  mem       REG                8,3   2135784  99890376 /home/bitcoin/testnet3/blocks/index/006671.ldb
bitcoind 2862 root  mem       REG                8,3   2136821  99890375 /home/bitcoin/testnet3/blocks/index/006670.ldb
bitcoind 2862 root  mem       REG                8,3   2136765  99890374 /home/bitcoin/testnet3/blocks/index/006669.ldb
bitcoind 2862 root  mem       REG                8,3   2138046  99890373 /home/bitcoin/testnet3/blocks/index/006668.ldb
bitcoind 2862 root  mem       REG                8,3    498092  99877353 /home/bitcoin/testnet3/chainstate/019082.ldb
bitcoind 2862 root  mem       REG                8,3    605726  99877351 /home/bitcoin/testnet3/chainstate/019080.ldb
bitcoind 2862 root  mem       REG                8,3    629112  99877350 /home/bitcoin/testnet3/chainstate/019079.ldb
bitcoind 2862 root  mem       REG                8,3   2135945  99890372 /home/bitcoin/testnet3/blocks/index/006667.ldb
bitcoind 2862 root  mem       REG                8,3   2136313  99890371 /home/bitcoin/testnet3/blocks/index/006666.ldb
bitcoind 2862 root  mem       REG                8,3    559747  99877349 /home/bitcoin/testnet3/chainstate/019078.ldb
bitcoind 2862 root  mem       REG                8,3    443670  99877348 /home/bitcoin/testnet3/chainstate/019077.ldb
bitcoind 2862 root  mem       REG                8,3    558150  99877326 /home/bitcoin/testnet3/chainstate/019074.ldb
bitcoind 2862 root  mem       REG                8,3    494770  99877289 /home/bitcoin/testnet3/chainstate/019068.ldb
bitcoind 2862 root  mem       REG                8,3   2135877  99890365 /home/bitcoin/testnet3/blocks/index/006665.ldb
bitcoind 2862 root  mem       REG                8,3    561643  99877284 /home/bitcoin/testnet3/chainstate/019066.ldb
bitcoind 2862 root  mem       REG                8,3    601259  99877275 /home/bitcoin/testnet3/chainstate/019062.ldb
bitcoind 2862 root  mem       REG                8,3    667092  99877274 /home/bitcoin/testnet3/chainstate/019061.ldb
bitcoind 2862 root  mem       REG                8,3   2137878  99890313 /home/bitcoin/testnet3/blocks/index/006664.ldb
bitcoind 2862 root  mem       REG                8,3   2137619  99890256 /home/bitcoin/testnet3/blocks/index/006663.ldb
bitcoind 2862 root  mem       REG                8,3   2135200  99890964 /home/bitcoin/testnet3/blocks/index/006709.ldb
bitcoind 2862 root  mem       REG                8,3   2136640  99890249 /home/bitcoin/testnet3/blocks/index/006662.ldb
bitcoind 2862 root  mem       REG                8,3   2138864  99890963 /home/bitcoin/testnet3/blocks/index/006708.ldb
bitcoind 2862 root  mem       REG                8,3   2137613  99890248 /home/bitcoin/testnet3/blocks/index/006661.ldb
bitcoind 2862 root  mem       REG                8,3   2137101  99890247 /home/bitcoin/testnet3/blocks/index/006660.ldb
bitcoind 2862 root  mem       REG                8,3   2136169  99889599 /home/bitcoin/testnet3/blocks/index/006580.ldb
bitcoind 2862 root  mem       REG                8,3   2137039  99890245 /home/bitcoin/testnet3/blocks/index/006659.ldb
bitcoind 2862 root  mem       REG                8,3   2135933  99890243 /home/bitcoin/testnet3/blocks/index/006658.ldb
bitcoind 2862 root  mem       REG                8,3   2135700  99891216 /home/bitcoin/testnet3/blocks/index/000217.ldb
bitcoind 2862 root  mem       REG                8,3   2136490  99890238 /home/bitcoin/testnet3/blocks/index/006657.ldb
bitcoind 2862 root  mem       REG                8,3   2135904  99891380 /home/bitcoin/testnet3/blocks/index/000216.ldb
bitcoind 2862 root  mem       REG                8,3   2137118  99890228 /home/bitcoin/testnet3/blocks/index/006656.ldb
bitcoind 2862 root  mem       REG                8,3   2137139  99890221 /home/bitcoin/testnet3/blocks/index/006655.ldb
bitcoind 2862 root  mem       REG                8,3   2136018  99891284 /home/bitcoin/testnet3/blocks/index/000215.ldb
bitcoind 2862 root  mem       REG                8,3   2136959  99890215 /home/bitcoin/testnet3/blocks/index/006654.ldb
bitcoind 2862 root  mem       REG                8,3   2138516  99890214 /home/bitcoin/testnet3/blocks/index/006653.ldb
bitcoind 2862 root  mem       REG                8,3   2136307  99891164 /home/bitcoin/testnet3/blocks/index/000214.ldb
bitcoind 2862 root  mem       REG                8,3   2136791  99890201 /home/bitcoin/testnet3/blocks/index/006652.ldb
bitcoind 2862 root  mem       REG                8,3   2135915  99891467 /home/bitcoin/testnet3/blocks/index/000213.ldb
bitcoind 2862 root  mem       REG                8,3   2136885  99890200 /home/bitcoin/testnet3/blocks/index/006651.ldb
bitcoind 2862 root  mem       REG                8,3   2136738  99890194 /home/bitcoin/testnet3/blocks/index/006650.ldb
bitcoind 2862 root  mem       REG                8,3   2135963  99891252 /home/bitcoin/testnet3/blocks/index/000212.ldb
bitcoind 2862 root  mem       REG                8,3   2136683  99890187 /home/bitcoin/testnet3/blocks/index/006649.ldb
bitcoind 2862 root  mem       REG                8,3   2136826  99890185 /home/bitcoin/testnet3/blocks/index/006648.ldb
bitcoind 2862 root  mem       REG                8,3   2136199  99891447 /home/bitcoin/testnet3/blocks/index/000211.ldb
bitcoind 2862 root  mem       REG                8,3   2137974  99890144 /home/bitcoin/testnet3/blocks/index/006647.ldb
bitcoind 2862 root  mem       REG                8,3   2136211  99891420 /home/bitcoin/testnet3/blocks/index/000210.ldb
bitcoind 2862 root  mem       REG                8,3   2138113  99890125 /home/bitcoin/testnet3/blocks/index/006646.ldb
bitcoind 2862 root  mem       REG                8,3   2136442  99889997 /home/bitcoin/testnet3/blocks/index/006645.ldb
bitcoind 2862 root  mem       REG                8,3   2135951  99891227 /home/bitcoin/testnet3/blocks/index/000209.ldb
bitcoind 2862 root  mem       REG                8,3   2136821  99889976 /home/bitcoin/testnet3/blocks/index/006644.ldb
bitcoind 2862 root  mem       REG                8,3   2136714  99889975 /home/bitcoin/testnet3/blocks/index/006643.ldb
bitcoind 2862 root  mem       REG                8,3   2136053  99877309 /home/bitcoin/testnet3/blocks/index/021496.ldb
bitcoind 2862 root  mem       REG                8,3   2136096  99891275 /home/bitcoin/testnet3/blocks/index/000208.ldb
bitcoind 2862 root  mem       REG                8,3   2136037  99877308 /home/bitcoin/testnet3/blocks/index/021495.ldb
bitcoind 2862 root  mem       REG                8,3   2136186  99877305 /home/bitcoin/testnet3/blocks/index/021494.ldb
bitcoind 2862 root  mem       REG                8,3   2137831  99889973 /home/bitcoin/testnet3/blocks/index/006642.ldb
bitcoind 2862 root  mem       REG                8,3   2135819  99877301 /home/bitcoin/testnet3/blocks/index/021492.ldb
bitcoind 2862 root  mem       REG                8,3   2136108  99877300 /home/bitcoin/testnet3/blocks/index/021491.ldb
bitcoind 2862 root  mem       REG                8,3   2135969  99891281 /home/bitcoin/testnet3/blocks/index/000106.ldb
bitcoind 2862 root  mem       REG                8,3   2136108  99889972 /home/bitcoin/testnet3/blocks/index/006641.ldb
bitcoind 2862 root  mem       REG                8,3   2136250  99877299 /home/bitcoin/testnet3/blocks/index/021490.ldb
bitcoind 2862 root  mem       REG                8,3   2135964  99877286 /home/bitcoin/testnet3/blocks/index/021489.ldb
bitcoind 2862 root  mem       REG                8,3   2136340  99877285 /home/bitcoin/testnet3/blocks/index/021488.ldb
bitcoind 2862 root  mem       REG                8,3   2136310  99889971 /home/bitcoin/testnet3/blocks/index/006640.ldb
bitcoind 2862 root  mem       REG                8,3   2135922  99877273 /home/bitcoin/testnet3/blocks/index/021487.ldb
bitcoind 2862 root  mem       REG                8,3    215722  99890481 /home/bitcoin/testnet3/blocks/index/006675.ldb
bitcoind 2862 root  mem       REG                8,3   1446810  99877303 /home/bitcoin/testnet3/blocks/index/021493.ldb
bitcoind 2862 root  mem       REG                8,3   1071297  99877269 /home/bitcoin/testnet3/blocks/index/021486.ldb
bitcoind 2862 root  mem       REG                8,3   2136043  99891174 /home/bitcoin/testnet3/blocks/index/000105.ldb
bitcoind 2862 root  mem       REG                8,3   2136904  99889970 /home/bitcoin/testnet3/blocks/index/006639.ldb
bitcoind 2862 root  mem       REG                8,3   2142388  99877258 /home/bitcoin/testnet3/blocks/index/021483.ldb
bitcoind 2862 root  mem       REG                8,3   1545966  99877257 /home/bitcoin/testnet3/blocks/index/021482.ldb
bitcoind 2862 root  mem       REG                8,2     22928    917083 /lib/x86_64-linux-gnu/libnss_dns-2.24.so
bitcoind 2862 root  mem       REG                8,3   2137078  99889969 /home/bitcoin/testnet3/blocks/index/006638.ldb
bitcoind 2862 root  mem       REG                8,3   1747373  99891228 /home/bitcoin/testnet3/blocks/index/000104.ldb
bitcoind 2862 root  mem       REG                8,3    649470  99877266 /home/bitcoin/testnet3/chainstate/019059.ldb
bitcoind 2862 root  mem       REG                8,3   2140557  99877255 /home/bitcoin/testnet3/blocks/index/021481.ldb
bitcoind 2862 root  mem       REG                8,3   1728515  99876973 /home/bitcoin/testnet3/blocks/index/021480.ldb
bitcoind 2862 root  mem       REG                8,3   2142815  99876932 /home/bitcoin/testnet3/blocks/index/021479.ldb
bitcoind 2862 root  mem       REG                8,3   2135893  99889968 /home/bitcoin/testnet3/blocks/index/006637.ldb
bitcoind 2862 root  mem       REG                8,3    579566  99877267 /home/bitcoin/testnet3/chainstate/019060.ldb
bitcoind 2862 root  mem       REG                8,3    548580  99877259 /home/bitcoin/testnet3/chainstate/019057.ldb
bitcoind 2862 root  mem       REG                8,3   2135394  99891347 /home/bitcoin/testnet3/blocks/index/000103.ldb
bitcoind 2862 root  mem       REG                8,3   2136837  99889957 /home/bitcoin/testnet3/blocks/index/006636.ldb
bitcoind 2862 root  mem       REG                8,3   2137290  99889956 /home/bitcoin/testnet3/blocks/index/006635.ldb
bitcoind 2862 root  mem       REG                8,3   2135457  99891282 /home/bitcoin/testnet3/blocks/index/000102.ldb
bitcoind 2862 root  mem       REG                8,3   2136564  99889949 /home/bitcoin/testnet3/blocks/index/006634.ldb
bitcoind 2862 root  mem       REG                8,3   2138394  99889947 /home/bitcoin/testnet3/blocks/index/006633.ldb
bitcoind 2862 root  mem       REG                8,3   2136030  99891441 /home/bitcoin/testnet3/blocks/index/000100.ldb
bitcoind 2862 root  mem       REG                8,3   2136735  99889945 /home/bitcoin/testnet3/blocks/index/006632.ldb
bitcoind 2862 root  mem       REG                8,3   2137542  99889943 /home/bitcoin/testnet3/blocks/index/006631.ldb
bitcoind 2862 root  mem       REG                8,3   2135721  99891449 /home/bitcoin/testnet3/blocks/index/000099.ldb
bitcoind 2862 root  mem       REG                8,3   2136354  99889932 /home/bitcoin/testnet3/blocks/index/006630.ldb
bitcoind 2862 root  mem       REG                8,3   2135690  99891093 /home/bitcoin/testnet3/blocks/index/000098.ldb
bitcoind 2862 root  mem       REG                8,3    586886  99877254 /home/bitcoin/testnet3/chainstate/019056.ldb
bitcoind 2862 root  mem       REG                8,3    615990  99877253 /home/bitcoin/testnet3/chainstate/019055.ldb
bitcoind 2862 root  mem       REG                8,3    571455  99877250 /home/bitcoin/testnet3/chainstate/019053.ldb
bitcoind 2862 root  mem       REG                8,3   2135703  99889931 /home/bitcoin/testnet3/blocks/index/006629.ldb
bitcoind 2862 root  mem       REG                8,3   2137312  99889927 /home/bitcoin/testnet3/blocks/index/006628.ldb
bitcoind 2862 root  mem       REG                8,3   2135304  99891163 /home/bitcoin/testnet3/blocks/index/000096.ldb
bitcoind 2862 root  mem       REG                8,3   2137016  99889924 /home/bitcoin/testnet3/blocks/index/006627.ldb
bitcoind 2862 root  mem       REG                8,3   2136814  99889923 /home/bitcoin/testnet3/blocks/index/006626.ldb
bitcoind 2862 root  mem       REG                8,3   2135902  99891171 /home/bitcoin/testnet3/blocks/index/000095.ldb
bitcoind 2862 root  mem       REG                8,3   2136330  99889921 /home/bitcoin/testnet3/blocks/index/006625.ldb
bitcoind 2862 root  mem       REG                8,3   2137377  99889920 /home/bitcoin/testnet3/blocks/index/006624.ldb
bitcoind 2862 root  mem       REG                8,3   2135655  99891167 /home/bitcoin/testnet3/blocks/index/000094.ldb
bitcoind 2862 root  mem       REG                8,3   2135930  99889918 /home/bitcoin/testnet3/blocks/index/006623.ldb
bitcoind 2862 root  mem       REG                8,3   2135798  99891269 /home/bitcoin/testnet3/blocks/index/000092.ldb
bitcoind 2862 root  mem       REG                8,3   2136814  99889916 /home/bitcoin/testnet3/blocks/index/006622.ldb
bitcoind 2862 root  mem       REG                8,3    504105  99877265 /home/bitcoin/testnet3/chainstate/019058.ldb
bitcoind 2862 root  mem       REG                8,3    588520  99877249 /home/bitcoin/testnet3/chainstate/019052.ldb
bitcoind 2862 root  mem       REG                8,3    606325  99877246 /home/bitcoin/testnet3/chainstate/019051.ldb
bitcoind 2862 root  mem       REG                8,3   2137399  99889912 /home/bitcoin/testnet3/blocks/index/006621.ldb
bitcoind 2862 root  mem       REG                8,3   2135504  99891469 /home/bitcoin/testnet3/blocks/index/000091.ldb
bitcoind 2862 root  mem       REG                8,3   2136501  99889909 /home/bitcoin/testnet3/blocks/index/006620.ldb
bitcoind 2862 root  mem       REG                8,3   2137438  99889906 /home/bitcoin/testnet3/blocks/index/006619.ldb
bitcoind 2862 root  mem       REG                8,3   2135912  99891280 /home/bitcoin/testnet3/blocks/index/000090.ldb
bitcoind 2862 root  mem       REG                8,3   2137631  99889900 /home/bitcoin/testnet3/blocks/index/006618.ldb
bitcoind 2862 root  mem       REG                8,3   2136659  99889892 /home/bitcoin/testnet3/blocks/index/006617.ldb
bitcoind 2862 root  mem       REG                8,3   2135525  99891070 /home/bitcoin/testnet3/blocks/index/000088.ldb
bitcoind 2862 root  mem       REG                8,3   2136814  99889865 /home/bitcoin/testnet3/blocks/index/006616.ldb
bitcoind 2862 root  mem       REG                8,3   2136363  99889861 /home/bitcoin/testnet3/blocks/index/006615.ldb
bitcoind 2862 root  mem       REG                8,3   2135776  99891339 /home/bitcoin/testnet3/blocks/index/000087.ldb
bitcoind 2862 root  mem       REG                8,3   2136792  99889852 /home/bitcoin/testnet3/blocks/index/006614.ldb
bitcoind 2862 root  mem       REG                8,3   2135986  99891445 /home/bitcoin/testnet3/blocks/index/000086.ldb
bitcoind 2862 root  mem       REG                8,3   2135870  99889833 /home/bitcoin/testnet3/blocks/index/006613.ldb
bitcoind 2862 root  mem       REG                8,3   2136296  99889808 /home/bitcoin/testnet3/blocks/index/006612.ldb
bitcoind 2862 root  mem       REG                8,3   1742483  99891041 /home/bitcoin/testnet3/blocks/index/000085.ldb
bitcoind 2862 root  mem       REG                8,3   2136917  99889800 /home/bitcoin/testnet3/blocks/index/006611.ldb
bitcoind 2862 root  mem       REG                8,3   1458075  99892142 /home/bitcoin/testnet3/blocks/index/007259.ldb
bitcoind 2862 root  mem       REG                8,3   2137012  99889795 /home/bitcoin/testnet3/blocks/index/006610.ldb
bitcoind 2862 root  mem       REG                8,3   2136739  99889793 /home/bitcoin/testnet3/blocks/index/006609.ldb
bitcoind 2862 root  mem       REG                8,3    785364  99877208 /home/bitcoin/testnet3/chainstate/019050.ldb
bitcoind 2862 root  mem       REG                8,3    626559  99877204 /home/bitcoin/testnet3/chainstate/019049.ldb
bitcoind 2862 root  mem       REG                8,3    619293  99877202 /home/bitcoin/testnet3/chainstate/019048.ldb
bitcoind 2862 root  mem       REG                8,3   2136704  99889790 /home/bitcoin/testnet3/blocks/index/006608.ldb
bitcoind 2862 root  mem       REG                8,3    457077  99877280 /home/bitcoin/testnet3/chainstate/019063.ldb
bitcoind 2862 root  mem       REG                8,3    484919  99877201 /home/bitcoin/testnet3/chainstate/019047.ldb
bitcoind 2862 root  mem       REG                8,3    666410  99877198 /home/bitcoin/testnet3/chainstate/019046.ldb
bitcoind 2862 root  mem       REG                8,3    478433  99877137 /home/bitcoin/testnet3/chainstate/019045.ldb
bitcoind 2862 root  mem       REG                8,3   2136658  99889787 /home/bitcoin/testnet3/blocks/index/006607.ldb
bitcoind 2862 root  mem       REG                8,3   2136759  99889786 /home/bitcoin/testnet3/blocks/index/006606.ldb
bitcoind 2862 root  mem       REG                8,3    334035  99877352 /home/bitcoin/testnet3/chainstate/019081.ldb
bitcoind 2862 root  mem       REG                8,3   2135957  99889781 /home/bitcoin/testnet3/blocks/index/006605.ldb
bitcoind 2862 root  mem       REG                8,3   2137974  99889772 /home/bitcoin/testnet3/blocks/index/006604.ldb
bitcoind 2862 root  mem       REG                8,3     49828  99877347 /home/bitcoin/testnet3/chainstate/019076.ldb
bitcoind 2862 root  mem       REG                8,3     37754  99877346 /home/bitcoin/testnet3/chainstate/019075.ldb
bitcoind 2862 root  mem       REG                8,3     40612  99877325 /home/bitcoin/testnet3/chainstate/019073.ldb
bitcoind 2862 root  mem       REG                8,3     42384  99877314 /home/bitcoin/testnet3/chainstate/019072.ldb
bitcoind 2862 root  mem       REG                8,3     41863  99877294 /home/bitcoin/testnet3/chainstate/019071.ldb
bitcoind 2862 root  mem       REG                8,3     39658  99877293 /home/bitcoin/testnet3/chainstate/019070.ldb
bitcoind 2862 root  mem       REG                8,3   1258343  99877136 /home/bitcoin/testnet3/chainstate/018401.ldb
bitcoind 2862 root  mem       REG                8,3   2136515  99889771 /home/bitcoin/testnet3/blocks/index/006603.ldb
bitcoind 2862 root  mem       REG                8,3   2136656  99889770 /home/bitcoin/testnet3/blocks/index/006602.ldb
bitcoind 2862 root  mem       REG                8,3     30143  99877376 /home/bitcoin/testnet3/chainstate/019085.ldb
bitcoind 2862 root  mem       REG                8,3     43866  99877291 /home/bitcoin/testnet3/chainstate/019069.ldb
bitcoind 2862 root  mem       REG                8,3     38250  99877288 /home/bitcoin/testnet3/chainstate/019067.ldb
bitcoind 2862 root  mem       REG                8,3   2137564  99889765 /home/bitcoin/testnet3/blocks/index/006601.ldb
bitcoind 2862 root  mem       REG                8,3     43770  99877283 /home/bitcoin/testnet3/chainstate/019065.ldb
bitcoind 2862 root  mem       REG                8,3    202673  99891517 /home/bitcoin/testnet3/chainstate/010403.ldb
bitcoind 2862 root  mem       REG                8,3   2139610  99889764 /home/bitcoin/testnet3/blocks/index/006600.ldb
bitcoind 2862 root  mem       REG                8,3     42660  99877251 /home/bitcoin/testnet3/chainstate/019054.ldb
bitcoind 2862 root  mem       REG                8,3    761005  99877139 /home/bitcoin/testnet3/chainstate/018402.ldb
bitcoind 2862 root  mem       REG                8,3   2137226  99889749 /home/bitcoin/testnet3/blocks/index/006599.ldb
bitcoind 2862 root  mem       REG                8,3   2137010  99889746 /home/bitcoin/testnet3/blocks/index/006598.ldb
bitcoind 2862 root  mem       REG                8,3     20720  99877005 /home/bitcoin/testnet3/indexes/txindex/036707.ldb
bitcoind 2862 root  mem       REG                8,3     88154  99876999 /home/bitcoin/testnet3/chainstate/019044.ldb
bitcoind 2862 root  mem       REG                8,3    293730  99891850 /home/bitcoin/testnet3/chainstate/009544.ldb
bitcoind 2862 root  mem       REG                8,3   2136380  99889738 /home/bitcoin/testnet3/blocks/index/006597.ldb
bitcoind 2862 root  mem       REG                8,3   2137195  99889734 /home/bitcoin/testnet3/blocks/index/006596.ldb
bitcoind 2862 root  mem       REG                8,3    105558  99876950 /home/bitcoin/testnet3/chainstate/019043.ldb
bitcoind 2862 root  mem       REG                8,3    125317  99876908 /home/bitcoin/testnet3/chainstate/019042.ldb
bitcoind 2862 root  mem       REG                8,3    109595  99876905 /home/bitcoin/testnet3/chainstate/019041.ldb
bitcoind 2862 root  mem       REG                8,3   1818268  99892162 /home/bitcoin/testnet3/blocks/index/007225.ldb
bitcoind 2862 root  mem       REG                8,3   2136826  99892161 /home/bitcoin/testnet3/blocks/index/007224.ldb
bitcoind 2862 root  mem       REG                8,3   2136691  99892160 /home/bitcoin/testnet3/blocks/index/007223.ldb
bitcoind 2862 root  mem       REG                8,3   2136293  99892159 /home/bitcoin/testnet3/blocks/index/007222.ldb
bitcoind 2862 root  mem       REG                8,3   2136486  99892158 /home/bitcoin/testnet3/blocks/index/007221.ldb
bitcoind 2862 root  mem       REG                8,3   2136574  99892157 /home/bitcoin/testnet3/blocks/index/007220.ldb
bitcoind 2862 root  mem       REG                8,3   2136766  99892156 /home/bitcoin/testnet3/blocks/index/007219.ldb
bitcoind 2862 root  mem       REG                8,3   2137041  99892155 /home/bitcoin/testnet3/blocks/index/007218.ldb
bitcoind 2862 root  mem       REG                8,3   2136735  99892154 /home/bitcoin/testnet3/blocks/index/007217.ldb
bitcoind 2862 root  mem       REG                8,3   2136359  99891025 /home/bitcoin/testnet3/blocks/index/007216.ldb
bitcoind 2862 root  mem       REG                8,3   2175576  99876926 /home/bitcoin/testnet3/chainstate/018400.ldb
bitcoind 2862 root  mem       REG                8,3   2179424  99891509 /home/bitcoin/testnet3/chainstate/009673.ldb
bitcoind 2862 root  mem       REG                8,3     56077  99877264 /home/bitcoin/testnet3/blocks/index/021484.ldb
bitcoind 2862 root  mem       REG                8,3    292996  99892153 /home/bitcoin/testnet3/blocks/index/007215.ldb
bitcoind 2862 root  mem       REG                8,3     13579  99877282 /home/bitcoin/testnet3/chainstate/019064.ldb
bitcoind 2862 root  mem       REG                8,2    178192    155975 /usr/lib/x86_64-linux-gnu/libevent_core-2.0.so.5.1.9
bitcoind 2862 root  mem       REG                8,2     14640    917058 /lib/x86_64-linux-gnu/libdl-2.24.so
bitcoind 2862 root  mem       REG                8,2     31744    917205 /lib/x86_64-linux-gnu/librt-2.24.so
bitcoind 2862 root  mem       REG                8,2   1689360    917089 /lib/x86_64-linux-gnu/libc-2.24.so
bitcoind 2862 root  mem       REG                8,2    135440    917180 /lib/x86_64-linux-gnu/libpthread-2.24.so
bitcoind 2862 root  mem       REG                8,2     92584    917073 /lib/x86_64-linux-gnu/libgcc_s.so.1
bitcoind 2862 root  mem       REG                8,2   1063328    917218 /lib/x86_64-linux-gnu/libm-2.24.so
bitcoind 2862 root  mem       REG                8,2   1566168    133375 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
bitcoind 2862 root  mem       REG                8,2    293264    155966 /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9
bitcoind 2862 root  mem       REG                8,2     10008    155987 /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.0.so.5.1.9
bitcoind 2862 root  mem       REG                8,2   2686672    133428 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
bitcoind 2862 root  mem       REG                8,2   1828312    151957 /usr/lib/libdb_cxx-4.8.so
bitcoind 2862 root  mem       REG                8,2     27048    154823 /usr/lib/x86_64-linux-gnu/libboost_chrono.so.1.62.0
bitcoind 2862 root  mem       REG                8,2    163240    154831 /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.62.0
bitcoind 2862 root  mem       REG                8,2    101624    133358 /usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.62.0
bitcoind 2862 root  mem       REG                8,2     14512    133848 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.62.0
bitcoind 2862 root  mem       REG                8,2    153288    917042 /lib/x86_64-linux-gnu/ld-2.24.so
bitcoind 2862 root  mem       REG                8,3       671  99877378 /home/bitcoin/testnet3/chainstate/019087.ldb
bitcoind 2862 root  mem       REG                8,3     35476  99892186 /home/bitcoin/testnet3/blocks/index/007313.ldb
bitcoind 2862 root  mem       REG                8,3      3477  99892093 /home/bitcoin/testnet3/blocks/index/007345.ldb
bitcoind 2862 root  mem       REG                8,3      3503  99892193 /home/bitcoin/testnet3/blocks/index/007408.ldb
bitcoind 2862 root  mem       REG                8,3       443  99892112 /home/bitcoin/testnet3/blocks/index/007440.ldb
bitcoind 2862 root  mem       REG                8,3     11730  99892125 /home/bitcoin/testnet3/blocks/index/007595.ldb
bitcoind 2862 root  mem       REG                8,3     21719  99876980 /home/bitcoin/testnet3/blocks/index/007629.ldb
bitcoind 2862 root  mem       REG                8,3      6512  99877102 /home/bitcoin/testnet3/blocks/index/009054.ldb
bitcoind 2862 root  mem       REG                8,3       582  99877022 /home/bitcoin/testnet3/blocks/index/009087.ldb
bitcoind 2862 root  mem       REG                8,3      2547  99877004 /home/bitcoin/testnet3/blocks/index/009120.ldb
bitcoind 2862 root  mem       REG                8,3      2381  99877092 /home/bitcoin/testnet3/blocks/index/010235.ldb
bitcoind 2862 root  mem       REG                8,3       445  99877125 /home/bitcoin/testnet3/blocks/index/010268.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877110 /home/bitcoin/testnet3/blocks/index/010454.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877112 /home/bitcoin/testnet3/blocks/index/010583.ldb
bitcoind 2862 root  mem       REG                8,3       713  99877132 /home/bitcoin/testnet3/blocks/index/010800.ldb
bitcoind 2862 root  mem       REG                8,3       308  99877131 /home/bitcoin/testnet3/blocks/index/011055.ldb
bitcoind 2862 root  mem       REG                8,3       444  99877142 /home/bitcoin/testnet3/blocks/index/011685.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877127 /home/bitcoin/testnet3/blocks/index/014328.ldb
bitcoind 2862 root  mem       REG                8,3       987  99877165 /home/bitcoin/testnet3/blocks/index/015657.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877138 /home/bitcoin/testnet3/blocks/index/015810.ldb
bitcoind 2862 root  mem       REG                8,3       305  99877166 /home/bitcoin/testnet3/blocks/index/016175.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877170 /home/bitcoin/testnet3/blocks/index/016478.ldb
bitcoind 2862 root  mem       REG                8,3       305  99877104 /home/bitcoin/testnet3/blocks/index/016785.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877176 /home/bitcoin/testnet3/blocks/index/017034.ldb
bitcoind 2862 root  mem       REG                8,3       307  99877189 /home/bitcoin/testnet3/blocks/index/017474.ldb
bitcoind 2862 root  mem       REG                8,3       307  99877164 /home/bitcoin/testnet3/blocks/index/017758.ldb
bitcoind 2862 root  mem       REG                8,3       305  99877168 /home/bitcoin/testnet3/blocks/index/018042.ldb
bitcoind 2862 root  mem       REG                8,3       304  99877178 /home/bitcoin/testnet3/blocks/index/019548.ldb
bitcoind 2862 root  mem       REG                8,2   1679776    132753 /usr/lib/locale/locale-archive
bitcoind 2862 root  mem       REG                8,3      2248  99877292 /home/bitcoin/testnet3/blocks/index/020202.ldb
bitcoind 2862 root  mem       REG                8,3      2541  99877256 /home/bitcoin/testnet3/blocks/index/020269.ldb
bitcoind 2862 root  mem       REG                8,3       584  99877323 /home/bitcoin/testnet3/blocks/index/020364.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877287 /home/bitcoin/testnet3/blocks/index/021118.ldb
bitcoind 2862 root  mem       REG                8,3       307  99877268 /home/bitcoin/testnet3/blocks/index/021308.ldb
bitcoind 2862 root  mem       REG                8,3       306  99877281 /home/bitcoin/testnet3/blocks/index/021341.ldb
bitcoind 2862 root  mem       REG                8,2     26258    133578 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
bitcoind 2862 root    0u      CHR                1,3       0t0      1029 /dev/null
bitcoind 2862 root    1u      CHR                1,3       0t0      1029 /dev/null
bitcoind 2862 root    2u      CHR                1,3       0t0      1029 /dev/null
bitcoind 2862 root    3uW     REG                8,3         0  99877725 /home/bitcoin/testnet3/.lock
bitcoind 2862 root    4w      REG                8,3  10471265  99879150 /home/bitcoin/testnet3/debug.log
bitcoind 2862 root    5u  a_inode               0,10         0      2054 [eventpoll]
bitcoind 2862 root    6u     unix 0xffff8848bca2c400       0t0 401418105 type=STREAM
bitcoind 2862 root    7u     unix 0xffff8848bca2c800       0t0 401418106 type=STREAM
bitcoind 2862 root    8u  a_inode               0,10         0      2054 [eventfd]
bitcoind 2862 root    9u     IPv6          401418109       0t0       TCP localhost:18332 (LISTEN)
bitcoind 2862 root   10u     IPv4          401418111       0t0       TCP localhost:18332 (LISTEN)
bitcoind 2862 root   11uW     REG                8,3         0  99879963 /home/bitcoin/testnet3/blocks/index/LOCK
bitcoind 2862 root   12w      REG                8,3   4073303  99877200 /home/bitcoin/testnet3/blocks/index/021477.log
bitcoind 2862 root   13w      REG                8,3     19429  99877203 /home/bitcoin/testnet3/blocks/index/MANIFEST-021475
bitcoind 2862 root   14u     IPv4          401423878       0t0       TCP kimsufi1:59326->158.140.212.157:18333 (ESTABLISHED)
bitcoind 2862 root   15uW     REG                8,3         0  99889659 /home/bitcoin/testnet3/chainstate/LOCK
bitcoind 2862 root   16w      REG                8,3       130  99877341 /home/bitcoin/testnet3/chainstate/019039.log
bitcoind 2862 root   17w      REG                8,3     61661  99877342 /home/bitcoin/testnet3/chainstate/MANIFEST-019037
bitcoind 2862 root   18uW     REG                8,3         0  99890497 /home/bitcoin/testnet3/indexes/txindex/LOCK
bitcoind 2862 root   19w      REG                8,3     11582  99877021 /home/bitcoin/testnet3/indexes/txindex/036708.log
bitcoind 2862 root   20w      REG                8,3    160759  99877108 /home/bitcoin/testnet3/indexes/txindex/MANIFEST-036706
bitcoind 2862 root   21uW     REG                8,3         0  99890745 /home/bitcoin/testnet3/.walletlock
bitcoind 2862 root   22w      REG                8,3         0  99879654 /home/bitcoin/testnet3/db.log
bitcoind 2862 root   23u      REG                8,3   1630208  99889541 /home/bitcoin/testnet3/wallet.dat
bitcoind 2862 root   24u      REG                8,3   1048576  99876927 /home/bitcoin/testnet3/database/log.0000000001
bitcoind 2862 root   25u     IPv4          401423978       0t0       TCP kimsufi1:45314->47.96.188.26:18333 (ESTABLISHED)
bitcoind 2862 root   26u  a_inode               0,10         0      2054 [eventpoll]
bitcoind 2862 root   27u     unix 0xffff88485733f400       0t0 401421983 type=STREAM
bitcoind 2862 root   28u     unix 0xffff88485733fc00       0t0 401421984 type=STREAM
bitcoind 2862 root   29u  a_inode               0,10         0      2054 [eventfd]
bitcoind 2862 root   30u     IPv6          401421986       0t0       TCP *:18333 (LISTEN)
bitcoind 2862 root   31u     IPv4          401421987       0t0       TCP *:18333 (LISTEN)
bitcoind 2862 root   32u     IPv4          401422908       0t0       TCP kimsufi1:37568->77.105.216.41:18333 (ESTABLISHED)
bitcoind 2862 root   33u     IPv6          401426464       0t0       TCP [2001:41d0:a:f115::1]:39958->nl.redfox.bz:18333 (ESTABLISHED)
bitcoind 2862 root   34u     IPv4          401422909       0t0       TCP kimsufi1:46234->ec2-34-229-185-163.compute-1.amazonaws.com:18333 (ESTABLISHED)

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Jan 31, 2019

So, you improved things with this branch. The wallet itself is completely released, and this does deserve to be committed. Now, I suspect the opening of these wallets needed blocks to be opened for rescanning of transactions, and that these blocks have not been released. This is going to be a more difficult patch now to release blocks needed by an unloaded wallet, and not by other wallets that may be loaded.

Thanks a lor for being patient.

@HashUnlimited

This comment has been minimized.

Copy link
Contributor

@HashUnlimited HashUnlimited commented Feb 1, 2019

this log reads too familiar for not attracting my attention....

for the basics (and I suppose that's what @sipa was after in his comment) unix based operating systems typically keep the contents of the memory until requested by another call. that's what's giving us the look-and-feel we are used to.

so fighting the symptoms like it appears here might not be the best solution.

so why familiar? we were looking at similar lines few months back and started tracing the issue. out of laziness we didn't finish the trace as we couldn't reproduce it on any other kernel/compiler combination than ours.

findings on troubleshooting: while configuring our compiler, we're asking it if it understands and can handle certain thingies like position independent code (PIC). typically, we will receive a "yes we can" answer. in the previous IF clause, the script decided to let the compiler apply the flags on demand by getting them passed through CXXFLAGS. unless we're on windows. ironically, gcc decides against PIC, at least on the crypto libraries, we didn't explicitly test against all (but bench would be a hot candidate)

@fscemama

This comment has been minimized.

Copy link
Author

@fscemama fscemama commented Feb 2, 2019

I think that after the #15297 hack, Bitcoin Core will be more stable when used in multiwallets context. I mean: the more memory you have, the more different wallets you should be able to use, given you load/unload without never reaching the ~300 simultaneous loaded wallets (unless you tweak BerlekeyDB settings and kernel semaphores, supposing this would work). Please, commit this to next official release, and I'll give you a real-world, production-context feedback.
@HashUnlimited : thank you for this analysis. I've always thought Bitcoin Core had been thought for a one-wallet use, and that BerkeleyDB had been chosen for easier cross-platform porting. I'm dreaming of a Bitcoin Core that would use an instance of Postgresql rather than BerkeleyDB, on a bitcoind setting. Why not after all.

meshcollider added a commit that referenced this issue Feb 5, 2019
…:Close

d3bf3b9 qa: Test .walletlock file is closed (João Barbosa)
2f8b8f4 wallet: Close wallet env lock file (João Barbosa)
8602a1e wallet: Close dbenv error file db.log (João Barbosa)

Pull request description:

  This PR closes `db.log` and removes `.walletlock` files when `BerkeleyEnvironment` is closed.

  Fixes #15291 (comment).

Tree-SHA512: 05d8b027feea914e0ba873e75d117857473d1fd7b400e41bd473d638171fa39d5be048990bf685dc0807f7d92418579b763056dc2a6dcf6b96777d5688ddee04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.