Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bitcoind 'compacts' chainstate database almost every time I run it #9684

Open
dooglus opened this issue Feb 4, 2017 · 7 comments

Comments

Projects
None yet
5 participants
@dooglus
Copy link
Contributor

commented Feb 4, 2017

I'm running the current master branch version (1c2edd9) and find that most times when I start bitcoind it spends about 10 minutes thrashing my hard drive doing a 'compaction' of the chainstate database.

The relevant stack trace:

#9  0x000055555596ce1f in leveldb::DBImpl::DoCompactionWork (this=0x7fffd8619e90, compact=0x7fffa8015860) at leveldb/db/db_impl.cc:1006
#10 0x000055555596d69f in leveldb::DBImpl::BackgroundCompaction (this=0x7fffd8619e90) at leveldb/db/db_impl.cc:736
#11 0x000055555596e192 in leveldb::DBImpl::BackgroundCall (this=0x7fffd8619e90) at leveldb/db/db_impl.cc:674
#12 0x00005555559951a3 in BGThread (this=0x7fffd847c490) at leveldb/util/env_posix.cc:582
#13 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7fffd847c490) at leveldb/util/env_posix.cc:521
#14 0x00007ffff4cc30a4 in start_thread (arg=0x7fffb73bc700) at pthread_create.c:309

I tried restarting bitcoind 10 times and kept a copy of the LOG file each time. Here are the file sizes:

-rw------- 1 chris chris 184366 Feb  3 11:19 LOG.00
-rw------- 1 chris chris 182024 Feb  3 11:30 LOG.01
-rw------- 1 chris chris    344 Feb  3 12:11 LOG.02
-rw------- 1 chris chris 194758 Feb  3 12:27 LOG.03
-rw------- 1 chris chris    344 Feb  3 12:27 LOG.04
-rw------- 1 chris chris 187248 Feb  3 12:36 LOG.05
-rw------- 1 chris chris    571 Feb  3 15:28 LOG.06
-rw------- 1 chris chris 189728 Feb  3 15:45 LOG.07
-rw------- 1 chris chris 194434 Feb  3 16:00 LOG.08
-rw------- 1 chris chris 188599 Feb  3 16:16 LOG.09

7 out of 10 times it did a full recompact, causing the computer to almost come to a standstill while it kept the harddrive busy. One of the 10 times bitcoind crashed hard almost immediately, but I'm sure that's a separate issue (#9683).

Here's the LOG file from the ~/.bitcoin/chainstate/ folder.

Edit: note that I use:

./configure --with-incompatible-bdb

And see "Using BerkeleyDB version Berkeley DB 5.3.28: (September 9, 2013)" in the debug.log.

If I run configure without that flag I see an error:

configure: error: Found Berkeley DB other than 4.8, required for portable wallets (--with-incompatible-bdb to ignore or --disable-wallet to disable wallet functionality)
@TheBlueMatt

This comment has been minimized.

Copy link
Contributor

commented Feb 4, 2017

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Feb 4, 2017

I used to be able to start bitcoind without it thrashing my hard drive for 10 minutes. And sometimes I still can (3 out of 10 times in my recent tests). Did it always read and rewrite most of the chainstate db on startup?

@sipa

This comment has been minimized.

Copy link
Member

commented Feb 4, 2017

I think it's related to how much has been written to the database before shutdown, and how long that was. With larger dbcache values in recent versions (in 0.14/master it can be up to 600M by default, if the mempool is empty), there may be a large dump to disk of dirty entries at shutdown, which need compaction at startup.

@sipa

This comment has been minimized.

Copy link
Member

commented Feb 4, 2017

@dooglus What version/options were you using last where you did not notice this behaviour?

@dooglus

This comment has been minimized.

Copy link
Contributor Author

commented Feb 4, 2017

I think it's related to how much has been written to the database before shutdown, and how long that was.

I was seeing a full recompact even when I had a fully synced blockchain and restarted shortly after starting up. ie. I'd start bitcoind, wait 10 minutes for the recompact to finish, then stop and restart it, and I'd see another full recompact immediately.

What version/options were you using last where you did not notice this behaviour?

I don't know. I didn't know this was a behavior that could happen until I first noticed it happening. These are the settings I use now, when it does happen:

keypool=10
logips=1
minrelaytxfee=0.0001
paytxfee=0.0001
upnp=0
spendzeroconfchange=0
logips=1
bind=127.0.0.1
maxconnections=32
testnet=0
txindex=1
checkblocks=1
listen=0
server=1
daemon=1

I changed to daemon=0 when testing with bitcoind instead of bitcoin-cli, but it didn't seem to change the behavior.

When I first noticed the behavior I was running from commit 453bda6. In the past I have run bitcored from bitpay, which comes with its own bitcoind executable, Bitcoin version v0.11.2.0-bc99ef3 (2016-04-08 10:42:17 -0400). I suppose that could be statically linked with a different leveldb version, and may be contributing to the behavior I'm seeing, since I run it on the same ~/.bitcoin/ directory.

I'll can rebuilding from older commits and see if I can determine which commit introduced the behavior if that would be helpful.

@sipa

This comment has been minimized.

Copy link
Member

commented Feb 4, 2017

Well the compaction will always run at startup, I believe, and always has. The question is whether it takes a long time.

v0.11.2 had a much smaller dbcache, and did much more frequent writes to disk. Also, txindex may affect performance (though that's in the block index db, not the chainstate).

@eklitzke

This comment has been minimized.

Copy link
Member

commented Mar 10, 2018

There are some things we can do to improve this but they all require modifying our in-tree copy of the LevelDB source code. That's something I'd like to do anyway, but I'm not sure everyone else is convinced that's a good idea.

@dooglus In the meantime you can use ionice to deprioritize the compaction thread. On Linux you can set this to a specific thread id. Usually the LevelDB compaction thread is the highest numbered thread id in the output of ps -eL. (Renaming that thread is another thing I'd like to change in the LevelDB source code).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.