Corruption: block checksum mismatch

[jorisbontje]

with bitcoind 0.11.0 on a 64bit Linux VPS (with simfs)

2015-08-06 05:53:59 Corruption: block checksum mismatch
2015-08-06 05:53:59 *** System error while flushing: Database corrupted
2015-08-06 05:53:59 Error: Error: A fatal internal error occured, see debug.log for details
2015-08-06 05:53:59 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-08-06 05:53:59 opencon thread interrupt
2015-08-06 05:53:59 addcon thread interrupt
2015-08-06 05:53:59 msghand thread interrupt
2015-08-06 05:53:59 scheduler thread interrupt
2015-08-06 05:53:59 net thread interrupt
2015-08-06 05:53:59 Shutdown: In progress...
2015-08-06 05:54:00 StopNode()
2015-08-06 05:54:00 Corruption: block checksum mismatch
2015-08-06 05:54:00 *** System error while flushing: Database corrupted
2015-08-06 05:54:00 Error: Error: A fatal internal error occured, see debug.log for details
2015-08-06 05:54:00 Shutdown: done

This is occasionally happening after a -reindex on 0.11.0, as this was also previously happening on earlier releases. bitcoind.conf includes settings preventing transaction flooding:

minrelaytxfee=0.00005
limitfreerelay=5

[laanwj]

These are disk corruption issues: leveldb keeps its own checksums of all data it writes, when it notices during (background) compaction that these mismatch it will return the error on the next flush.

2015-08-06 05:53:59 Corruption: block checksum mismatch
2015-08-06 05:53:59 *** System error while flushing: Database corrupted

[Weax]

Same issue here, but with Win10 x64. Checkdisk returned no errors on disk.

[jorisbontje]

The support team of the VPS hosting provider claims there are no disk corruption issues:

There is no disk corruption from our side, because all our nodes use RAID array in which case
disks is duplicated in the array and in case of any disk failure this disk is automatically removed
from the array and node is working with other left disk until faulty hard disk replace.
You database corruption may be related to your program/service crash or errors in it etc.

[TheBlueMatt]

Silent corruption in raid arrays is absolutely possible. Its possible this is an issue with leveldb, but it seems more likely there are silent errors somewhere in your disk/raid controller/etc.

[rnicoll]

Have seen something similar in dogecoin/dogecoin#1216 - could be specific to our changes, but the version in question is a complete rebuild around Bitcoin Core 0.11. Either way, am looking into this, if we find anything we will of course push fixes into Bitcoin as well.

[sipa]

@jorisbontje Is it possible that you ran out of memory or the disk was full?

[jorisbontje]

@sipa enough disk space (50GB+). I was expecting an out of memory (only 3GB total), but didn't get an Out of memory error anywhere. I configured the relay settings to reduce the pending tx pool, but still happening.

I can try to reproduce this with a -reindex, is there any kind of additional monitoring / debug logging, etc you'd like me to enable?

[sipa]

Joris: does the crash happen while you are synchronizing/running, or after a stop/start? If so, could you try running with -par=1 (to reduce risk for CPU problems, but I don't think that is the problem here, you'd get validation errors instead). My last guess would be an incompatibility between LevelDB and the filesystem yo're using.

[jorisbontje]

This happened again while synchronizing/running bitcoind -reindex -par=1

2015-08-20 16:56:48 UpdateTip: new best=0000000000000000043a35f28c76f951eef172b33a53a49c47a891c759943341  height=370730  log2_work=83.236584  tx=80585198  date=2015-08-20 16:55:41 progress=0.999999  cache=62.6MiB(15705tx)
2015-08-20 16:56:48 UpdateTip: 1 of last 100 blocks above version 3
2015-08-20 16:56:48 keypool reserve 3695
2015-08-20 16:56:48 keypool return 3695
2015-08-20 16:56:49 Corruption: block checksum mismatch
2015-08-20 16:56:49 *** System error while flushing: Database corrupted
2015-08-20 16:56:49 Error: Error: A fatal internal error occured, see debug.log for details
2015-08-20 16:56:49 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-08-20 16:56:49 net thread interrupt
2015-08-20 16:56:49 opencon thread interrupt
2015-08-20 16:56:49 addcon thread interrupt
2015-08-20 16:56:49 scheduler thread interrupt
2015-08-20 16:56:49 msghand thread interrupt
2015-08-20 16:56:49 Shutdown: In progress...
2015-08-20 16:56:49 StopNode()
2015-08-20 16:56:50 Corruption: block checksum mismatch
2015-08-20 16:56:50 *** System error while flushing: Database corrupted
2015-08-20 16:56:50 Error: Error: A fatal internal error occured, see debug.log for details
2015-08-20 16:56:51 Shutdown: done

[Mikadily]

I'm experiencing similar problem. Ubuntu 12.04.5 on a physical box, software raid.

[miguelmorales85]

I have the same problem here. It all started with the following message "ERROR: AcceptToMemoryPool: non-final". Then after stopping bitcoind I get this message
"Corruption: block checksum mismatch
*** System error while flushing: Database corrupted"

[jorisbontje]

@miguelmorales85 what kind of filesystem / storage setup do you have? (raid, vps, simfs, etc)

[miguelmorales85]

@jorisbontje the hd was formated in ext4

[chuacw]

I have the same issue. and I don't have disk corruption.
My database was newly imported from bootstrap.dat

This is Bitcoin Core 0.11 (64-bit), and on Windows 2008 R2 (64-bit), NTFS. I have about 1TB free.

2015-09-14 08:42:33 UpdateTip: new best=0000000000000000184d37ea91dc37ad61f2c9eca556020476787bf9916a5a24 height=337433 log2_work=81.923139 tx=55737117 date=2015-01-04 11:54:53 progress=0.635931 cache=62.6MiB(10952tx)
2015-09-14 08:42:34 Corruption: block checksum mismatch
2015-09-14 08:42:34 *** System error while flushing: Database corrupted
2015-09-14 08:44:13 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-09-14 08:44:13 opencon thread interrupt
2015-09-14 08:44:13 scheduler thread interrupt
2015-09-14 08:44:13 addcon thread interrupt
2015-09-14 08:44:13 net thread interrupt
2015-09-14 08:44:14 ERROR: AcceptToMemoryPool: non-final
2015-09-14 08:44:14 msghand thread interrupt
2015-09-14 08:44:14 Shutdown: In progress...
2015-09-14 08:44:14 StopNode()
2015-09-14 08:44:14 UPNP_DeletePortMapping() returned: 0
2015-09-14 08:44:14 upnp thread interrupt
2015-09-14 08:44:14 Corruption: block checksum mismatch
2015-09-14 08:44:14 *** System error while flushing: Database corrupted
2015-09-14 08:44:16 Shutdown: done

[miguelmorales85]

Again database corruption at block 252.dat
I am reindexing.. this takes a while so I will let you know the results...

[miguelmorales85]

I have just failed to reindex bitcoind.. this is what happened.
2015-09-16 05:15:09 UpdateTip: new best=0000000000000000087d41de97e50f2c63151b0caee4a63f1387dd0eb8ae2153 height=345409 log2_work=82.326044 tx=60898518 date=2015-02-27 14:50:06 progress=0.724568 cache=64.9MiB(14575tx)
2015-09-16 05:15:09 Corruption: block checksum mismatch
2015-09-16 05:15:09 *** System error while flushing: Database corrupted
2015-09-16 05:15:10 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-16 05:15:10 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-09-16 05:15:10 addcon thread interrupt
2015-09-16 05:15:10 opencon thread interrupt
2015-09-16 05:15:10 scheduler thread interrupt
2015-09-16 05:15:10 net thread interrupt
2015-09-16 05:15:10 msghand thread interrupt
2015-09-16 05:15:10 Shutdown: In progress...
2015-09-16 05:15:10 RPCAcceptHandler: Error: Operation canceled
2015-09-16 05:15:10 StopNode()
2015-09-16 05:15:11 Corruption: block checksum mismatch
2015-09-16 05:15:11 *** System error while flushing: Database corrupted
2015-09-16 05:15:11 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-16 05:15:11 Shutdown: done

can somebody help me. Please dont tell me I have to reindex AGAIN...

[jgarzik]

@miguelmorales85 it is highly likely you have hardware problems

[miguelmorales85]

@jgarzik I ran e2fsk on the unmounted partition in a live session with 0 errors :(

[funyung]

Firstly, I created my account because of this issue.
Maybe this isn't a hardware issue, as I am also running x64 Linux (ubuntu 14.04) w/ no disk errors.
I've tried reindexing around 6 times since thursday. 5 out of those times I hit database corruption at random points, but before this last time it hit I had filled my ext4 drive when it hit 2 weeks til completion.
So I set up a symlink so the blocks directory is stored on a much larger ntfs drive. Go to reindex on next launch because it's required and I hit database corruption at 52 weeks. I literally just purchased the ntfs drive a month ago, and I've not had any errors reported on either disks.

[miguelmorales85]

I finally finally did not get the checksum mismatch after several reindexing (like a week of try and error). But now the node keeps getting STALLED, I have can not SYNC the node. The closer I got was like verification progress to 0.9989xxxx and I am slowly going down... Can somebody help me? I have restarted the node, reboot the Ubuntu PC and I just get the node UP and then STALLED. Of course I am accepting conections.

[sipa]

Miguel: anything in debug.log mentioning invalid blocks?

[miguelmorales85]

@sipa Hello, I have run "cat /home/USER/.bitcoin/debug.log | grep invalid"
and found this:
2015-09-26 02:50:28 ERROR: AcceptBlockHeader: block is marked invalid
2015-09-26 02:50:28 ERROR: invalid header received
2015-09-26 04:49:47 ERROR: AcceptBlockHeader: block is marked invalid
2015-09-26 04:49:47 ERROR: invalid header received
2015-09-26 07:25:55 ERROR: AcceptBlockHeader: block is marked invalid
2015-09-26 07:25:55 ERROR: invalid header received

what can I do about it?

[sipa]

That almost certainly means Bitcoin Core incorrectly decided that a signature or transaction was invalid in a block, and subsequently marked the block invalid. Can you try reindexing with -par=1 or par=1 in bitcoin.conf?

[miguelmorales85]

oh god... I feared you would say that... Ok .. will do. See you in a few days. Thanks.

[miguelmorales85]

Hello @sipa I have reindexed. I have the same problem, I can never get to SYNC the node. At this moment the node progression is at 0.9983xxxx and going down and down every seccond... it's going to Stall again. So no par=1 in bitcoin.conf did something to prevent my problem.

[santzi]

Hi,
I have got this several times. About 1-2 times per week. When I started bitcoind again, it working ok and I don't need to do reindex it again. So this is not so fatal, but annoying.
My Bitcoin core version is quite new. I compiled it from source (master) about 2-3 weeks ago. Now there are over 200 commits more than this version.
I think pure 0.11.0 version working better.

Here is the last log on yestersay:
2015-10-07 11:29:45 receive version message: /bitcoinseeder:0.01/: version 60000, blocks=230000, us=88.193.138.220:8333, peer=593
2015-10-07 11:32:39 receive version message: /Snoopy:0.2.1/: version 70001, blocks=0, us=88.193.138.220:8333, peer=594
2015-10-07 11:32:42 receive version message: /bitcoinj:0.14SNAPSHOT/: version 70001, blocks=377853, us=127.0.0.1:8333, peer=595
2015-10-07 11:33:20 Corruption: block checksum mismatch
2015-10-07 11:33:20 *** System error while flushing: Database corrupted
2015-10-07 11:33:20 Error: Error: A fatal internal error occurred, see debug.log for details
2015-10-07 11:33:20 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-10-07 11:33:21 scheduler thread interrupt
2015-10-07 11:33:21 opencon thread interrupt
2015-10-07 11:33:21 addcon thread interrupt
2015-10-07 11:33:21 msghand thread interrupt
2015-10-07 11:33:21 net thread interrupt
2015-10-07 11:33:21 Shutdown: In progress...
2015-10-07 11:33:21 StopNode()
2015-10-07 11:33:21 Corruption: block checksum mismatch
2015-10-07 11:33:21 *** System error while flushing: Database corrupted
2015-10-07 11:33:21 Error: Error: A fatal internal error occurred, see debug.log for details
2015-10-07 11:33:21 Shutdown: done

When this occurred next time, I'll pull new sources from master branch and try again.

[trevelocity]

With bitcoind 0.11.1 on a 64-bit Linux PC I got the "Corruption: block checksum mismatch" error.
I ran a full system check, which reported single bit data dependent corruption in one of the DIMMs.
I think the LevelDB data was corrupted when cached in the DRAM memory and then got written
back to disk. I replaced both DIMMs, ran with -reindex=1, and bitcoind has been running without
error for a week.

[z0rg1nc]

Have the same on Ubuntu 15.04 x64 + Bitcoin core 0.11.1.0 (VM 2GB ram, enough hard drive space, ram check OK). First it happened 2 days ago (24hr client just receiving info most of the time), I deleted last 3 blk filed, reindex started, on -2y30w throws "error reading from database". Yesterday I cleared btc data directory except bitcoin.conf and wallet.dat files, moved btc directory to different physical drive with zfs, after resync from scratch it throws "error reading from database" on -2y30w again.

2015-12-17 00:35:19 UpdateTip: new best=000000000000005bb239fc8ff69c2d9fd33d1d23532dcebcb24192758f1baba3  height=237055  log2_work=70.080825  tx=18133567  date=2013-05-20 12:51:09 progress=0.081912  cache=4.8MiB(3525tx)
2015-12-17 00:35:19 UpdateTip: new best=000000000000015767a84949b0bbf7ba25c24515e6198fd57909b8d3d1d7a7f5  height=237056  log2_work=70.08088  tx=18133568  date=2013-05-20 12:51:48 progress=0.081912  cache=4.8MiB(3526tx)
2015-12-17 00:35:19 UpdateTip: new best=0000000000000156d33717e7055beb017cd07e0d044fc7a4a364cf459d148aa1  height=237057  log2_work=70.080936  tx=18134049  date=2013-05-20 12:55:01 progress=0.081914  cache=5.0MiB(4225tx)
2015-12-17 00:35:20 UpdateTip: new best=00000000000000ce70375942e44e5a5f5f2192afecdc178fe0fe240bb8310d1c  height=237058  log2_work=70.080991  tx=18134844  date=2013-05-20 13:18:02 progress=0.081917  cache=5.9MiB(5339tx)
2015-12-17 00:35:20 UpdateTip: new best=00000000000001679e03844bc5e651ff9394b57f4612a43e555fc5cedab5c3b2  height=237059  log2_work=70.081047  tx=18135625  date=2013-05-20 13:51:19 progress=0.081921  cache=6.2MiB(6234tx)
2015-12-17 00:35:20 UpdateTip: new best=000000000000001a74e886007d45abf8feae6668516e35462fe207f0bfe48b7e  height=237060  log2_work=70.081102  tx=18135989  date=2013-05-20 14:02:32 progress=0.081923  cache=6.9MiB(7048tx)
2015-12-17 00:35:20 UpdateTip: new best=0000000000000027162a00ff456c8f1f2a37ac915714c17b9d2c622dab9b94e6  height=237061  log2_work=70.081158  tx=18136400  date=2013-05-20 14:03:09 progress=0.081924  cache=7.0MiB(7490tx)
2015-12-17 00:35:20 UpdateTip: new best=00000000000000f6f3787321ff929f44a9bfc0e75ac54ac90157e5bf5dd23cb6  height=237062  log2_work=70.081213  tx=18136619  date=2013-05-20 14:14:18 progress=0.081925  cache=7.4MiB(7899tx)
2015-12-17 00:35:20 UpdateTip: new best=0000000000000158fc5cee0bcac58973ddc158e4891df80eef7c67f78e5cd47e  height=237063  log2_work=70.081269  tx=18136747  date=2013-05-20 14:17:25 progress=0.081926  cache=7.8MiB(8135tx)
2015-12-17 00:35:20 LevelDB read failure: Corruption: block checksum mismatch
2015-12-17 00:35:20 Corruption: block checksum mismatch

2015-12-17 09:36:10 init message: Rescanning...
2015-12-17 09:36:10 Rescanning last 1101 blocks (from block 235950)...
2015-12-17 09:36:10  rescan                    1ms
2015-12-17 09:36:10 init message: Activating best chain...
2015-12-17 09:36:11 UpdateTip: new best=000000000000008bdabb35a3ce5f85b8c3a11ffefd5a29defcd4c786e271f8a9  height=237052  log2_work=70.080658  tx=18132299  date=2013-05-20 11:49:50 progress=0.081864  cache=1.2MiB(809tx)
2015-12-17 09:36:11 UpdateTip: new best=00000000000000b9f7b4bfaee1287b3e51fd534cdb30f42c482dfff7f714f6d3  height=237053  log2_work=70.080714  tx=18132304  date=2013-05-20 12:10:02 progress=0.081864  cache=1.3MiB(820tx)
2015-12-17 09:36:11 UpdateTip: new best=00000000000000afe58811aea548dac4d6f4fd8c9c2b88836c84170ab52b03b8  height=237054  log2_work=70.080769  tx=18133326  date=2013-05-20 12:35:09 progress=0.081869  cache=3.6MiB(3003tx)
2015-12-17 09:36:12 UpdateTip: new best=000000000000005bb239fc8ff69c2d9fd33d1d23532dcebcb24192758f1baba3  height=237055  log2_work=70.080825  tx=18133567  date=2013-05-20 12:51:09 progress=0.081870  cache=4.3MiB(3525tx)
2015-12-17 09:36:12 UpdateTip: new best=000000000000015767a84949b0bbf7ba25c24515e6198fd57909b8d3d1d7a7f5  height=237056  log2_work=70.08088  tx=18133568  date=2013-05-20 12:51:48 progress=0.081870  cache=4.3MiB(3526tx)
2015-12-17 09:36:12 UpdateTip: new best=0000000000000156d33717e7055beb017cd07e0d044fc7a4a364cf459d148aa1  height=237057  log2_work=70.080936  tx=18134049  date=2013-05-20 12:55:01 progress=0.081872  cache=4.5MiB(4225tx)
2015-12-17 09:36:12 UpdateTip: new best=00000000000000ce70375942e44e5a5f5f2192afecdc178fe0fe240bb8310d1c  height=237058  log2_work=70.080991  tx=18134844  date=2013-05-20 13:18:02 progress=0.081876  cache=5.4MiB(5339tx)
2015-12-17 09:36:12 UpdateTip: new best=00000000000001679e03844bc5e651ff9394b57f4612a43e555fc5cedab5c3b2  height=237059  log2_work=70.081047  tx=18135625  date=2013-05-20 13:51:19 progress=0.081879  cache=5.8MiB(6234tx)
2015-12-17 09:36:13 UpdateTip: new best=000000000000001a74e886007d45abf8feae6668516e35462fe207f0bfe48b7e  height=237060  log2_work=70.081102  tx=18135989  date=2013-05-20 14:02:32 progress=0.081881  cache=6.5MiB(7048tx)
2015-12-17 09:36:13 UpdateTip: new best=0000000000000027162a00ff456c8f1f2a37ac915714c17b9d2c622dab9b94e6  height=237061  log2_work=70.081158  tx=18136400  date=2013-05-20 14:03:09 progress=0.081883  cache=6.6MiB(7490tx)
2015-12-17 09:36:13 UpdateTip: new best=00000000000000f6f3787321ff929f44a9bfc0e75ac54ac90157e5bf5dd23cb6  height=237062  log2_work=70.081213  tx=18136619  date=2013-05-20 14:14:18 progress=0.081884  cache=7.0MiB(7899tx)
2015-12-17 09:36:13 UpdateTip: new best=0000000000000158fc5cee0bcac58973ddc158e4891df80eef7c67f78e5cd47e  height=237063  log2_work=70.081269  tx=18136747  date=2013-05-20 14:17:25 progress=0.081884  cache=7.3MiB(8135tx)
2015-12-17 09:36:13 LevelDB read failure: Corruption: block checksum mismatch
2015-12-17 09:36:13 Corruption: block checksum mismatch
2015-12-17 09:36:53 Error reading from database: Database corrupted

Several guys from bitcointalk.org faced the same problem too (https://bitcointalk.org/index.php?topic=1288944.0 https://bitcointalk.org/index.php?topic=1281530.0 https://bitcointalk.org/index.php?topic=1152996.0)

UPDATE: Looks like it's my hardware problem, cancelling bitcoin code claims.

[iuriguilherme]

I have this issue on a fresh install of debian jessie/sid.

It's happening with bitcoin, litecoin and dogecoin.

It happens on btrfs, changing filesystem to ext4 and redownloading the blochain helped.

[iuriguilherme]

I have looked at my logs and this probably has to do with changing the libdb version.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=621425

[jloughry]

I had the same problem on a Mac Mini (mid 2011) running OS X 10.11.2 with all updates installed, 8 GB RAM. It was consistent; neither -reindex nor deleting the entire blockchain and re-downloading would get around the corrupted database problem, no matter how many times I tried. However, replacing the 0.11.2 software with 0.10.3 (64 bit) worked, after I ran the 0.10.3 with -reindex. So there is something in 0.11.2 that doesn't like my machine, but I can work with 0.10.3 for now.

[ethought]

We are having the same problem and it is recurring. All other clients seem fine (Litecoin, Dogecoin, Unobtanium etc) - it seems to only be affecting bitcoin.

2016-01-22 16:30:51 Leaving block file 424: CBlockFileInfo(blocks=0, size=0, heights=0...0, time=1970-01-01...1970-01-01)
2016-01-22 16:30:56 Pre-allocating up to position 0x1000000 in blk00424.dat
2016-01-22 16:30:56 LevelDB read failure: Corruption: block checksum mismatch
2016-01-22 16:30:56 Corruption: block checksum mismatch
2016-01-22 16:30:56 Error: Error reading from database, shutting down.
2016-01-22 16:30:56 Error reading from database: Database corrupted

"version" : 110200,
"protocolversion" : 70002,
"walletversion" : 60000,

[iuriguilherme]

I have tried changing libdb versions on a debian sid machine and only bitcoin is being affected as of now. Dogecoin and every other wallet are fine.

[laanwj]

Changing libdb won't fix this: the backing database for the utxo set and block index is leveldb, not berkeleydb.

[cornwarecjp]

FYI: I had the same issue with v0.11.0 on AMD64 Debian Jessie Linux with ext4 on LUKS file system (Bitcoin-Qt compiled by me).

I ran memtest86 and found a couple of faults in my RAM. I disabled 2 sections of 64kB of misbehaving RAM in the GRUB configuration, restarted, and started re-indexing with Bitcoin. Re-indexing took almost a week on my computer; every 8 hours or so I stopped Bitcoin, made a backup of the databases and restarted(*) Bitcoin, just in case something went wrong halfway or at 99%, to prevent having to start re-indexing again. However, I did not encounter any problems during re-indexing, so for me, the problem is solved.

If you are always encountering issues during re-indexing on the same position in the block chain, could it be that there is something wrong in the block files stored on your hard disk? In that case, would it be possible to resolve the issue by removing a certain range of block files, to force Bitcoin to re-download these blocks? WARNING: this is just guess-work, and certainly not a recommendation! I'm not that much of an expert, and from my POV this might as well make things worse.

(*) On 2nd, 3rd etc. start of Bitcoin, I did not provide the re-index commandline argument, since I did not want to re-start indexing every time. Apparently, Bitcoin automatically continues re-indexing if re-indexing was unfinished on the last shutdown.

[giacecco]

Reindexing with -par=1 solved it for me on MacOS running 0.11.2.

The only interesting thing my side is that when I had the error for the first time I had just finished re-downloading the whole chain using 0.11.2, after upgrading from 0.11.1. I did not need to re-download, I just did it (it is not relevant why). This would seem to suggest it is a problem that - at least on Mac - arose from 0.11.2.

From people's descriptions though it seems to me we may be in front of more than one issue. The range of symptoms is wide and inconsistent.

G.

[lostcodder]

Same problem. 0.12.1 Win7 x64

2016-04-21 22:05:08 Corruption: block checksum mismatch
2016-04-21 22:05:08 *** System error while flushing: Database corrupted
2016-04-21 22:19:48 UPnP Port Mapping successful.
2016-04-21 22:22:15 ERROR: ProcessNewBlock: ActivateBestChain failed
2016-04-21 22:22:15 tor: Thread interrupt
2016-04-21 22:22:15 opencon thread interrupt
2016-04-21 22:22:15 net thread interrupt
2016-04-21 22:22:15 torcontrol thread exit
2016-04-21 22:22:15 addcon thread interrupt
2016-04-21 22:22:15 scheduler thread interrupt
2016-04-21 22:22:15 msghand thread interrupt
2016-04-21 22:22:15 Shutdown: In progress...
2016-04-21 22:22:15 StopNode()
2016-04-21 22:22:16 UPNP_DeletePortMapping() returned: 0
2016-04-21 22:22:16 upnp thread interrupt
2016-04-21 22:22:16 Corruption: block checksum mismatch
2016-04-21 22:22:16 *** System error while flushing: Database corrupted
2016-04-21 22:22:18 Shutdown: done

[netzstrand]

I have the same problem :(
Kubuntu 16.04 64bit
bitbitcoin-qt 0.12.1.0-g9779e1e

2016-05-16 05:40:43 Corruption: block checksum mismatch
2016-05-16 05:40:43 *** System error while flushing: Database corrupted
2016-05-16 05:57:02 ping timeout: 1200.046503s
2016-05-16 05:57:23 ping timeout: 1200.016167s
2016-05-16 05:59:52 ping timeout: 1200.020610s
2016-05-16 07:33:18 ERROR: ProcessNewBlock: ActivateBestChain failed
2016-05-16 07:33:18 tor: Thread interrupt
2016-05-16 07:33:18 torcontrol thread exit
2016-05-16 07:33:18 addcon thread interrupt
2016-05-16 07:33:18 opencon thread interrupt
2016-05-16 07:33:18 msghand thread interrupt
2016-05-16 07:33:18 scheduler thread interrupt
2016-05-16 07:33:18 net thread interrupt
2016-05-16 07:33:18 Shutdown: In progress...
2016-05-16 07:33:18 StopNode()
2016-05-16 07:33:18 Corruption: block checksum mismatch
2016-05-16 07:33:18 *** System error while flushing: Database corrupted
2016-05-16 07:33:30 Shutdown: done

[jloughry]

Update: this problem went away when I replaced the memory (RAM) in my computer. The old memory was found to have errors by memtest (data mismatch error). The new memory has no such errors according to memtest.

Bitcoin Core has been running 24/7 for weeks now with no problems on a Mac Mini (Mid 2011, 2.3 GHz Intel Core i5, 16 GB 1600 MHz DDR3 memory, OS X 10.11.4).

[whitslack]

For what it's worth, I've run Memtest86+ overnight for two nights and found no errors in my RAM, so that's not the only reason this error can arise.

Discovered that I was getting some very occasional single-bit errors when reading from disk, so that was likely the cause of this error for me. I underclocked my system by 3.76%, and now I am unable to reproduce the single-bit errors, so hopefully my "block checksum mismatch" issues will be gone now too.

[hamiltino]

I had the same error i fixed it by formatting my disk to f2fs from ext4.

[Leviathn]

This appears to be entirely related to hardware failures. Closeable @borisbontje

[Leviathn]

@jorisbontje