Fatal LevelDB error: Corruption: block checksum mismatch

[kristapsk]

Had this problem with one long running node. Tried to restart bitcoind with -debug=leveldb -reindex, ended up with:

2021-04-18T15:00:03Z UpdateTip: new best=0000000000000000001153fc4f4aee341eacc92b28450dfff90810b4d2df216e height=616461 version=0x2fffe000 log2_work=91.642167 tx=500946903 date='2020-02-08T04:25:07Z' progress=0.799912 cache=147.0MiB(649296txo)
2021-04-18T15:00:03Z leveldb: Generated table #215595: 44038 keys, 2177932 bytes
2021-04-18T15:00:04Z leveldb: Generated table #215596: 44998 keys, 2180094 bytes
2021-04-18T15:00:04Z leveldb: Generated table #215597: 23369 keys, 1204615 bytes
2021-04-18T15:00:04Z leveldb: Compacted 1@1 + 2@2 files => 5562641 bytes
2021-04-18T15:00:04Z leveldb: compacted to: files[ 0 10 124 455 1486 0 0 ]
2021-04-18T15:00:04Z leveldb: Delete type=2 #215401
2021-04-18T15:00:04Z leveldb: Delete type=2 #215292
2021-04-18T15:00:04Z leveldb: Delete type=2 #215579
2021-04-18T15:00:04Z leveldb: Delete type=2 #215580
2021-04-18T15:00:04Z leveldb: Compacting 1@2 + 0@3 files
2021-04-18T15:00:04Z LevelDB read failure: Corruption: block checksum mismatch
2021-04-18T15:00:04Z Fatal LevelDB error: Corruption: block checksum mismatch
2021-04-18T15:00:04Z You can use -debug=leveldb to get more complete diagnostic messages
2021-04-18T15:00:05Z leveldb: Generated table #215598: 28652 keys, 1426450 bytes
2021-04-18T15:00:05Z Error: Error reading from database, shutting down.
2021-04-18T15:00:05Z Error reading from database: Fatal LevelDB error: Corruption: block checksum mismatch

Any hints how to debug this?

# bitcoind -version
Bitcoin Core version v0.20.1.0-gentoo

[sipa]

Sounds like a hardware or OS problem.

[jarolrod]

@kristapsk I used to get this on an NVME drive that was overheating. Fixed that and haven't seen this since.

[kristapsk]

It's a Gigabyte Brix with SATA HDD with nothing strange in a dmesg output...

[jarolrod]

Was bitcoin compiled (i assume so since on Gentoo), try recompiling again? Also, make sure to run diagnostics on your system, there could be a hardware issue somewhere. If not with the HDD then maybe with memory?

This is marcofalke's template for these type of issues :)

Bitcoin Core makes heavy use of CPU, RAM, and disk IO. Hardware defects might only become visible when running Bitcoin Core. You might want to check your hardware for defects.

memtest86 to check your RAM
to check the CPU behavior under load, use linpack or Prime95
to test your storage device use smartctl or CrystalDiskInfo

Source: https://bitcoin.stackexchange.com/a/12206

Edit:
Below are some similar issues, the common theme is a hardware issue with HD, memory, or overheating CPU:

• LevelDB read failure: Corruption: block checksum mismatch #11355
• LevelDB read failure: Corruption: block checksum mismatch #13009
• Recurring LevelDB errors on 0.21.0rc1 #20455
• LevelDB read failure: Corruption: block checksum mismatch #10897

[kristapsk]

@jarolrod , thanks for the hints / links, will test the hardware.

[jarolrod]

@kristapsk any update?

[maflcko]

Closing for now

[kristapsk]

Did some hardware tests on a specific machine back in a May, haven't found any problems (but I don't remember details about everything I tested anymore). Was busy with other things, have decided to replace it with something more serious for mainnet. But will try to figure out what's wrong there at some point probably. It's ok to close this for now.

[ademar111190]

I had this checksum LevelDB problem frequently in the last weeks, following this answer #10897 (comment) I managed to solve it, the HD I have been using to keep a copy of bitcoins blockchain is not working properly anymore, copying everything to a new HD and using it completely solve the errors.

[tansanDOTeth]

I had this checksum LevelDB problem frequently in the last weeks, following this answer #10897 (comment) I managed to solve it, the HD I have been using to keep a copy of bitcoins blockchain is not working properly anymore, copying everything to a new HD and using it completely solve the errors.

I read that comment, but not sure what I have to do run tests against my external. Do you have more instructions written somewhere?