LevelDB read failure: Corruption: block checksum mismatch always happens at 250Mb cache threshold

[fabioganga]

Dear friends,

I am experiencing a quite weird issue when performing the initial sync with Bitcoin Core wallet 0.16.0 on my iMac.

The wallet always crashes at a specific point, exactly 250 Mb cache.

I have tried to increase the cache setting to the double of what is the default value, to 900 Mb to no avail.

debug.log shows that it ALWAYS gets stuck at exactly the same point, with the wallet showing the "Error reading from the database, shutting down" pop up.

Launching the wallet again syncs for a little while more until it gets stuck again at exactly the same point again.

Debug.log shows the following as the cause when crashing:

2018-03-07 02:06:20 UpdateTip: new best=000000000000000000f77a9ac3b079f9df4fbc1c01382e5da665d3f86123a7fa height=485013 version=0x20000002 log2_work=87.097193 tx=254082032 date='2017-09-13 11:40:48' progress=0.826117 cache=250.3MiB(1841606txo)
2018-03-07 02:06:20 LevelDB read failure: Corruption: block checksum mismatch
2018-03-07 02:06:20 Corruption: block checksum mismatch
2018-03-07 02:06:26 Error reading from database: Database corrupted

2018-03-07 02:08:39 UpdateTip: new best=000000000000000000f77a9ac3b079f9df4fbc1c01382e5da665d3f86123a7fa height=485013 version=0x20000002 log2_work=87.097193 tx=254082032 date='2017-09-13 11:40:48' progress=0.826116 cache=250.3MiB(1841606txo)
2018-03-07 02:08:39 LevelDB read failure: Corruption: block checksum mismatch
2018-03-07 02:08:39 Corruption: block checksum mismatch
2018-03-07 02:08:56 Error reading from database: Database corrupted

2018-03-07 02:28:11 UpdateTip: new best=000000000000000000f77a9ac3b079f9df4fbc1c01382e5da665d3f86123a7fa height=485013 version=0x20000002 log2_work=87.097193 tx=254082032 date='2017-09-13 11:40:48' progress=0.826105 cache=250.3MiB(1841606txo)
2018-03-07 02:28:11 LevelDB read failure: Corruption: block checksum mismatch
2018-03-07 02:28:11 Corruption: block checksum mismatch

These are the specs of my iMac: late 2012, i7 3.4 Ghz processor (quad core 8 threads) 32Gb RAM.

Thank you so much for your help, much appreciated.

[eklitzke]

Can you run with -debug=leveldb and provide more verbose error messages? I'm also interested in seeing a copy of the corrupted table file. The name of the corrupted file should be in the logs when you use -debug=leveldb

To actually proceed with syncing you'll need to run with -reindex to rebuild your local copy of the utxo database. This will take a long time because it has to reprocess old blocks, but it won't re-download any blocks you already have locally.

[fabioganga]

Hi @eklitzke and thanks for your interest in solving my issue. Apologies for the delay, I saved the entire folder containing the files producing this error so I could get back to it at a later stage but in the meanwhile I synced from scratch importing the private keys to a new wallet that is now running fine.

I have now forced the client to use this corrupt data directory because I hope that my experience will help solving the issue that could occur to others.

I have therefore run the client as you suggested with -debug=leveldb and here are the error messages obtained this time:

debug.log

db.log that is referenced to is totally blank, no information at all.

I can see that it makes a reference to .dat files, please let me know which one you would like to analyze and I will happily upload it for you.

Many thanks

[eklitzke]

I have some notes in #12690 about LeveLDB corruption, but this seems to be a separate issue. The error message "block checksum mismatch" implies that LevelDB handed out data with checksum X to the kernel, but then later read that data back with checksum Y. That could be bad hardware, a bad filesystem, or anywhere in-between. But I don't think it's a bug in Bitcoin (or LevelDB).

[udovdh]

If such things (levelDB issues) can happen on machines not overheating, not having faulty disks or faulty RAM, then why isn't bitcoind more resilient? Another charged particle can ruin your day...
Do we need to move to ECC and get some RAM-producer's stawks? Or will there be a software workaround?