Running 0.12rc01 with pruning. "Corrupted block database" now.

[murchandamus]

Started running 0.12rc01 a couple of days ago. As the host only has limited storage, I was running it in pruning mode. Checking on my node today, it is sporting a "Corrupted block database" now.

debug.log:

2016-01-20 09:15:39 ERROR: ApplyTxInUndo: undo data overwriting existing transaction
2016-01-20 09:15:39 ERROR: ApplyTxInUndo: undo data overwriting existing output
2016-01-20 09:15:40 ERROR: ApplyTxInUndo: undo data adding output to missing
2016-01-20 09:16:31 ERROR: VerifyDB(): *** coin database inconsistencies found (last 1 blocks, 0 good transactions before that)
2016-01-20 09:16:32 : Corrupted block database detected.

Unfortunately, debug log is missing everything older than an hour. Is there anything else I could provide?

[pstratem]

@xekyo Did you receive this error when running 0.12rc1 or 0.11.x ?

[murchandamus]

@pstratem 0.12rc1

[pstratem]

@xekyo what are the lines before this? (So I can take a guess at what was going on)

[murchandamus]

Unfortunately, only hundreds of lines repeating the above ERROR messages.

When I tried to restart the process I got this:

2016-01-20 09:35:02 Bitcoin version v0.12.0rc1 (2016-01-13 18:08:05 +0100)
2016-01-20 09:35:02 Prune configured to target 20480MiB on disk for block and undo files.
2016-01-20 09:35:02 Using OpenSSL version OpenSSL 1.0.1e 11 Feb 2013
2016-01-20 09:35:02 Default data directory /home/pi/.bitcoin
2016-01-20 09:35:02 Using data directory /home/pi/.bitcoin
2016-01-20 09:35:02 Using config file /home/pi/.bitcoin/bitcoin.conf
2016-01-20 09:35:02 Using at most 125 connections (1024 file descriptors available)
2016-01-20 09:35:02 Using 4 threads for script verification
2016-01-20 09:35:02 scheduler thread start
2016-01-20 09:35:02 libevent: getaddrinfo: address family for nodename not supported
2016-01-20 09:35:02 Binding RPC on address ::1 port 8332 failed.
2016-01-20 09:35:02 HTTP: creating work queue of depth 16
2016-01-20 09:35:02 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to us
e cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.HTTP: starting 4 worker
threads
2016-01-20 09:35:02 Error: Couldn't open socket for incoming connections (socket returned error Address family not supported by pr
otocol (97))
2016-01-20 09:35:02 Bound to 0.0.0.0:8333
2016-01-20 09:35:02 Cache configuration:
2016-01-20 09:35:02 * Using 2.0MiB for block index database
2016-01-20 09:35:02 * Using 20.0MiB for chain state database
2016-01-20 09:35:02 * Using 28.0MiB for in-memory UTXO set
2016-01-20 09:35:02 init message: Loading block index...
2016-01-20 09:35:02 Opening LevelDB in /home/pi/.bitcoin/blocks/index
2016-01-20 09:35:02 Opened LevelDB successfully
2016-01-20 09:35:02 Using obfuscation key for /home/pi/.bitcoin/blocks/index: 0000000000000000
2016-01-20 09:35:02 Opening LevelDB in /home/pi/.bitcoin/chainstate
2016-01-20 09:35:02 Opened LevelDB successfully
2016-01-20 09:35:02 Using obfuscation key for /home/pi/.bitcoin/chainstate: 0000000000000000
2016-01-20 09:35:29 LoadBlockIndexDB: last block file = 421
2016-01-20 09:35:29 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=87, size=70324481, heights=393971...394057, time
=2016-01-19...2016-01-19)
2016-01-20 09:35:29 Checking all blk files are present...
2016-01-20 09:35:29 LoadBlockIndexDB(): Block files have previously been pruned
2016-01-20 09:35:29 LoadBlockIndexDB: transaction index disabled
2016-01-20 09:35:30 LoadBlockIndexDB: hashBestChain=0000000000000000071f795753f5812d12305dbfda26303378a36b89ff33b15e height=394057
date=2016-01-19 20:40:59 progress=0.999572
2016-01-20 09:35:30 init message: Verifying blocks...
2016-01-20 09:35:30 Verifying last 288 blocks at level 3
2016-01-20 09:35:30 ERROR: DisconnectBlock(): added transaction mismatch? database corrupted
2016-01-20 09:35:30 ERROR: ApplyTxInUndo: undo data overwriting existing output

[murchandamus]

@pstratem Is there anything else I could provide or run to get more information?

[maflcko]

What is the memory usage of RAM and Storage?

[murchandamus]

@MarcoFalke: Since it won't run right now, I don't know how I would provide accurate numbers. I'm running this on a RaspberryPi 2B with 1 GiB RAM. I'm pruning to 20GiB, on a budget of 64GiB.

1.2G ./.bitcoin/chainstate
60M ./.bitcoin/blocks/index
21G ./.bitcoin/blocks
22G ./.bitcoin

[laanwj]

So to be clear: you were already using pruning in 0.11.x, without problems, but after running 0.12.0rc1 on the same data directory this problem occured?

Do you have a backup of your chainstate before the test with rc1?

[murchandamus]

laanwj: I was running 0.11.2 without problems with the same settings (pruning to 20GiB), I replaced it with 0.12.0rc1 on Monday evening (less than 48 hours ago), on the same directory.

I don't have a backup of the chainstate before the test unfortunately.

[pstratem]

@laanwj I've encountered the same issue running git master

[maflcko]

@pstratem Is your debug.log truncated as well? Did you save it?

[mrbandrews]

I tried to reproduce this error by running 0.11.2 on an old datadir (with pruning on) until Jan 18; then switched to 0.12.0rc1 and continued syncing. No problems here.
@pstratem When you say you have this issue running master, does that mean running master consistently (as opposed to, say, running 0.11 and then switching to master)?