LevelDB corruption with 0.8.x Mac client #2770
Comments
|
I am also still having this problem again with 0.8.2. Previously I had the problem with prior 0.8 builds and had to downgrade. 0.8.2 looked good for a while, but I've once again gotten this. |
|
@rebroad you're using a mac? |
|
I don't believe you've ever previously disclosed that you were running it on truecrypt. Can you reproduce your failures with it not on truecrypt? |
|
Bump. I am also getting this issue. OS X 0.8.3 |
|
I still think it's time to upgrade to the latest LevelDB code and see if that helps. |
|
... @Diapolo If a new version of LevelDB has changed in ways we don't understand, then those changes could result in chain forking. If we understand how its changed we could reason about how they may or may not have helped OSX/Windows and there is no need to just guess. |
|
@gmaxwell You are most likely correct in what you say, but doing nothing about the crashes seems to just bring people away from bitcoind or Bitcoin-Qt in the end :-/. |
|
I got a chainstate corruption this morning on my Mac, and have spent most of the day debugging. My MANIFEST- file is corrupted (corrupted file: http://skypaint.com/bitcoin/MANIFEST-076191 ). I wrote a little python tool to dump the log records (http://skypaint.com/bitcoin/dumplogfile.py ), and something odd happened at bytes 65,506-65.536: there are 30 zero bytes. So the records look like (output from my dumplogfile.py ): FULL length 1012 (position: 64490) ... and leveldb is very upset that there is no FIRST record before the LAST record at the beginning of block 2 (block as in "leveldb 32,768-byte block of records", not bitcoin block). WHY those 30 zero bytes were written.... I dunno. |
|
...or not written, as mmap'd pages often default to all-zero. Gah. |
|
This looks like the same issue: https://code.google.com/p/leveldb/issues/detail?id=197 |
|
@gavinandresen Did this happen with a build, which includes the recent LevelDB update? |
|
@Diapolo: yes, I got corruption running git HEAD which includes latest LevelDB. I've reached the edge of my LevelDB/filesystem knowledge, so I'm done debugging this for now and moving on to other things. |
|
@gavinandresen Were you able to reliably reproduce the corruption? |
|
Guys, thanks again for finally looking into this and acknowledging that something is seriously amiss. Given that it seems we have a fairly serious and profound bug to squash here, and we've apparently (momentarily?) reached the edge of your time/knowledge to debug this, would it make sense as a stopgap measure to release a backport v0.7.3 Mac client? https://bitcointalk.org/index.php?topic=199699.msg2128991#msg2128991 Windows and Linux versions were released back in May, even though their v0.8.x releases seem to work pretty good. From what I've seen, none of the v0.8.x Mac releases with LevelDB work reliably. I've made some noise after each of the v0.8.x Mac releases that they are not working reliably, but this is the first time I've seen any of the devs semi-publically acknowledge it. I'm sure there are plenty of Mac users downloading and running these buggy releases every day, with no idea there may be a problem, and the number of posts to the bitcointalk Tech Support forum about this confirm it. I am well-informed and technical enough to know to use v0.7.2 and install the "May 15 workaround", but I'm sure there are PLENTY of other Mac users who are not. A public announcement that the existing v0.8.x Mac releases are bad and an official release of a v0.7.3 Mac backport I think is probably the right next step. |
|
@toffoo I am not comfortable releasing a 0.7.3 final until the May15 hardfork has actually split from the older clients and the backport has proven reliable. I also don't have a Mac, so I cannot even build v0.7.3rc3 for it. |
|
@toffoo : I'd rather drop Bitcoin-Qt on OSX than recommend an old release; we don't have the resources to support multiple releases. Whether this bug is serious enough to drop the OSX release until somebody figures out what is wrong and fixes it is debatable-- I've had two corruptions in the last six months, the seriousness of this problem seems to be different depending either on particular hardware or luck (I have no idea which). |
|
Could this be a result of relying on fsync on Mac OSX for write guarantees? https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fsync.2.html
If so, the fix would be to use fcntl F_FULLFSYNC instead, although that can cause significant slowdown as it will flush all buffered data, not just that of the particular file. https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man2/fcntl.2.html
other references: If this is the cause, I believe the fix is a simple patch to leveldb's port/port_posix.h along the lines of:
|
|
I think we should try that patch. The fact that fsync doesn't fsync is ... unfortunate. I guess if writes can arrive to disk out of order and at some arbitrary later time could would line up with what I've seen which is that the corruptions only happen when something goes wrong at the OS level, like a kernel panic during resume from suspend. I will ask Sanjay to weigh in on the LevelDB bug as well. He is not a Mac expert but could probably provide advice. |
|
Please try: |
|
I gave 0.8.4 a try but I am still getting the same error. I am just one user though, and not a programmer, so better wait for another response to confirm. |
|
@medicinebottle Did you restart from scratch? (start with -reindex option, or wiped your blocks + chainstate directories). |
|
Hi, yes I would also like to try out this new test release, but I was curious which would be the most scientific method of indexing the block data:
In all my previous tests of v0.8.x releases I did method #1. For this one I was thinking of trying #2. Would that in any way ruin the scientific integrity of this test? How long should a complete sync of the blockchain thru bitcoin-qt take these days? |
|
As per sipa's suggestion, I've gone with #1 above and reindexed my existing v0.7.2 blockchain. Almost 3 days now running v0.8.4rc2 here and: zero corruption Thanks again guys for all this Mac client attention. Whatever you did seems to be working. |
|
@toffoo If you're up for another experiment to make sure the 0.8.4 changes are the cause here: can you try doing a reindex with 0.8.3 again, and see whether it fails? |
|
Okay, done. This afternoon I reindexed with 0.8.3 no problem and it has continued to run (on/off several times) tonight with no crashes or corruption. When I started this Issue it was 0.8.2 that had corrupted 6 times on me. I don't think I ever tried 0.8.3 when it came out, because I understood that nothing had changed that might address this issue, and I was fed up after all my problems with 0.8.0, 0.8.1, and 0.8.2. I will continue running both this 0.8.3 and 0.8.4rc2 on and off and will report back here if I can get either to corrupt or crash. |
|
::sigh:: we fixed a file descriptor exhaustion bug that caused corruption on OSX previously. |
|
Ok, experiment complete: 0.8.3 just corrupted on me. Reindexing worked fine and ran perfect for about 2 days, now upon restart today it opened with "Database corrupted", closed "unexpectedly", and didn't offer to reindex. Like before, this was the only interesting line in debug.log: LevelDB read failure: Corruption: block checksum mismatch I will go install the newly released 0.8.4 and report again here if this problem resurfaces again. |
|
Ok, now things are getting interesting. With the newly installed 0.8.4, first it came up saying my wallet.dat is corrupted (that's a first) but fortunately I have a backup copy. With restored wallet.dat in place, now it offers to re-index, but now actually crashes/corrupts during re-index (at least during the beginning!): LevelDB read failure: Corruption: block checksum mismatch So I've tried this a few times now and I cannot get it to re-index the crashed 0.8.3 blockchain. So now I will try again from scratch with a fresh copy of my good 0.7.2 blockchain. |
|
I just tried out the new 0.8.5 Mac release by reindexing my good 0.7.2 blockchain. It reindexed fine in about 3 hours and I left it running with no problems for several hours. Upon the first restart I got the "Corrupted block database detected. Do you want to rebuilt the block database now?" message and this (new!) error in debug.log: Opened LevelDB successfully |
|
@toffoo Any chance you can upload all your files except wallet.dat and bitcoin.conf? AFAIK, they are the only two files that can affect your privacy, but maybe someone else here can provide more information about that. Will really help with the debugging. |
|
Some updates.... After so many versions, tests, crashes, and discussions I'm bound to get dizzy and lose track of what's been going on, so here's a log of the latest: I torrented a fresh copy of @jgarzik's excellent bootstrap.dat and was successfully able to build a fresh reindex with the -OMG6b binary. Indexing above block height 250,000 still seemed to take a very long time, with all sorts of errors in debug.log, but did eventually complete and run successfully. All tests with -OMG6b have so far worked fine (with regards to the levelDB corruption issue) but a different issue was encountered: directly after sending a transaction (and using Coin Control for my first time) bitcoin-qt crashed with an OSX traceback and here it is: This same issue was also reported by LitecoinDev coblee. I was able to restart bitcoin-qt with no DB corruption reported and the tx that caused the crash did go thru. Apart from this probably unrelated issue, -OMG6b (from the bootstrap.dat reindex) continues to work fine and has not produced or reported corruption. @rescrv has requested a core dump from a DB corruption crash (if possible). We determined that OSX does not default to save core dumps, but it can be turned on (before the crash happens of course) with the command: ulimit -c unlimited The Bitcoin-Qt binary must then be run from the same Terminal using the command: /Applications/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt This setting is not robust to reboots or new Terminal windows and must therefore always be run beforehand each time. I successfully generated a core dump by reproducing the 'send tx' bug. And we also determined that it would probably be a security risk to share a core dump from bitcoin-qt running a "live wallet". Currently under discussion whether I continue testing with -OMG6b, revert to -OMG5, or await a new binary? |
|
https://bitcointalk.org/index.php?topic=337294.msg3718821#msg3718821 http://pastebin.com/hg2QTwTB The crash during Send is unrelated to the leveldb issue. It seems to be in Coin Control. |
|
Update: Running v0.8.6-mactest1 worked fine for me last night and this morning, opened/closed many times and transactions sent with no problem ... until upon a restart today I got the blockchain database corruption error. debug.log: chainstate/: Also noticed this version no longer has Retina font support, all the previous OMG builds did. |
|
As per @rescrv 's suggestion, I ran a memtest to discount a bad RAM hypothesis. Thanks edubai on the forum for Mac memtest link: https://bitcointalk.org/index.php?topic=337294.msg3764745#msg3764745 I passed: |
|
Hi @wtogami, Bitcoin-Qt-0.8.5-OMG7-no-mmap OSX crashed for me upon Sending a transaction using Coin Control, here's the pastebin: ...but running good so far with no LevelDB corruption. |
|
@toffoo |
|
Can we separate the GUI bugs from the OS X LevelDB corruption bug? The issue that was reported resulted in a very clear pattern of corruption, where contiguous segments of files written by LevelDB would be erroneously replaced by NULL bytes. Since applying the msync patch I provided, the corruption issue originally reported has not been confirmed by anyone else, including toffoo (the original reporter). What's been reported since has been a series of hard crashes, mostly within Qt code, that do not follow the pattern of corruption originally reported in any of the linked forum posts. These bugs reside outside LevelDB, and are not part of the (now infamous) LevelDB corruption bug. |
|
@rescrv Toffoo's crashes at startup appear to be database corruption related, though perhaps not the same corruption. (The GUI crashes are related to some other unrelated code that was backported and put in warren's "OMG" builds, and indeed don't belong here. though it's good to have learned about them) |
|
Toffoo's crashes at startup are most likely because the state is invalid, yes. But that does not imply that it is LevelDB corruption. Every time he has had corruption, it has been preceded by a crash in code that is not LevelDB. In the first chainstate he posted since the crash, there was absolutely zero LevelDB corruption. Every file is parseable in its entirety. Every entry is retrievable using any access pattern (iteration or "get"). Every checksum matches the blocks it should. If the state is corrupt, it was corrupted before being put into LevelDB, or (and this is unlikely), the corruption affected the data and the checksums just right so that everything is consistent. The problem he reports here is at a higher level, and was preceded by the crash in Qt. In the second chainstate he posted (mactest1), there is indeed corruption in The underlying cause of the GUI crashes is likely also responsible for whatever bug within Bitcoin-Qt is now corrupting toffoo's state. But that bug is not the LevelDB bug that was first reported where the data was corrupted on a clean shutdown. The recent reports have been linked to the GUI, and the nature of the crash would also directly explain the subsequent corruption as well. According to @wtogami on IRC, the GUI code was "a simple race condition in the GUI code, probably a double free", which is exactly the kind of bug that would lead to errant corruption that indiscriminately affects data both inside and outside of LevelDB. |
|
I am sorry the unrelated GUI bug confused the issue here, but I think we are passed that. The corruption during compaction occurred with the mactest1 build, which lacks the buggy GUI code in question. https://github.com/bitcoin/bitcoin/commits/0.8.6 |
|
https://bitcointalk.org/index.php?topic=337294.msg3718821#msg3718821 The "mactest1" build is also there which instead uses @rescrv's #3327 mmap fix. |
|
A careful reading of the OS X manpage for msync indicates that it may not provide the same guarantees as linux. "The msync() system call writes modified whole pages back to the filesystem and updates the file modification time." This would seem to indicated that the pages are flushed to the filesystem layer as dirty pages rather than flushed to disk as the linux manpage indicates (and the actual implementation guarantees). Importantly PosixMmapFile::Sync calls fdatasync before calling msync. This could result in corruption at anypoint in the file assuming Sync is only called once when the file is closed for writing (as I believe nearly all of the files created with the WriteableFile class are). It would be realllly helpful if we could find an apple engineer to verify the actual functioning of msync(). |
|
Okay, Bitcoin-Qt-0.8.5-OMG7-no-mmap2 also worked good for me with no corruption but today I switched to 0.8.5-OMG8. |
|
Glad to hear. Bitcoin-Qt-0.8.5-OMG7-no-mmap2 and Bitcoin-Qt-0.8.5-OMG8 were effectively the same thing, the latter was just properly tagged in git. =) |
|
I downloaded Litecoin QT for the very first time yesterday to a Macbook Pro Retina, received my first litecoins just now and immediately afterwards had this error. Now Litecoin QT won't open without the error closing it and I have no idea what to do. I had just received LTC and had no time to backup the wallet. Is there a laymen/noob course of action to take? |
|
@mrgitbit |
|
Is this ticket still open, since 0.8.6 has been released? |
ghost
mentioned this issue
|
Still have this issue, Bitcoin 0.9.3 on Ubuntu 12.04.5 LTS 64bit, FS is ext4 |
|
The issue can be resolved by updating leveldb to the latest one. |
|
Recently I've found it was actually bad RAM. |
|
Thanks for letting us know what the problem was. Heh yes, Bitcoin Core is good in finding any problems with your RAM and CPU, usually one of the first programs that starts failing because it requires all results to be correct and deterministic. One bit flip can be enough. Crypto is fragile like that. In a way the early warning is good - stop using the machine for bitcoin until you are sure it is fixed, corruption in wallet operations could result in coin loss. |
… instead of mempool (bitcoin#2770) * Don't rely on UTXO set in CheckCanLock The UTXO set only works for TXs in the mempool and won't work when we try to retroactively lock unlocked TXs from blocks. This is safe as ProcessTx is only called when a TX was accepted into the mempool or connected in a block, which means that all input checks were good. * Rename RetryLockMempoolTxs to RetryLockTxs and let it retry connected TXs * Instead of manually calling ProcessTx, let SyncTransaction handle all cases SyncTransaction is called from AcceptToMemoryPool and when transactions got connected in a block. So this is the time we want to run TXs through ProcessTx. This also enables retroactive signing of TXs that were unknown before a new block appeared. * Test retroactive signing and safe TXs in LLMQ ChainLocks tests * Also test for retroactive signing of chained TXs * Honor lockedParentTx when looking for TXs to retry signing * Stop scanning for TXs to retry after a depth of 6 * Generate 6 block to avoid retroactive signing overloading Travis * Avoid retroactive signing * Don't rely on NewPoWValidBlock and use SyncTransaction to build blockTxs NewPoWValidBlock is not guaranteed to be called when blocks come in fast. When a block is accepted in AcceptBlock, NewPoWValidBlock is only called when the new block is a successor of the currently active tip. This is not the case when after the first block a second block is accepted immediately as the first block is not connected yet. This might be a bug actually in the handling of NewPoWValidBlock, so we might need to check/fix this later, but currently I prefer to not touch that part. Instead, we now use SyncTransaction to gather TXs for blockTxs. This works because SyncTransaction is called for all transactions in a freshly connected block in one go. The call also happens before UpdatedBlockTip is called, so it's fine with the existing logic. * Use tx.IsCoinBase() instead of checking index 0 Also check for empty vin.
and others
I've run bitcoin-qt Mac client on the latest OSX version since 0.3.x thru to 0.7.2 with literally zero problems ever, on essentially this same hardware setup.
Since the 0.8 update and switch to LevelDB none of the three Mac client releases have worked stably for me and I've had to downgrade to 0.7.2 (with the May 15 workaround) to maintain a stable bitcoin wallet.
Current setup is: MacBook Pro Retina, OSX 10.8.4, bitcoin-qt 0.8.2
I saw that some of the known corruption issues/bugs were fixed/closed with the 0.8.2 release, so I decided to try the upgrade again. After re-indexing and working fine for hours at time (much better than 0.8.1 at least) upon restart I get "Error opening block database. Do you want to rebuild the block database?" which of course I don't want to do because it takes forever, even on this world's fastest Mac with SSD drive. This has happened 6 times now, and the interesting line in debug.log is:
Verifying last 288 blocks at level 3
LevelDB read failure: Corruption: block checksum mismatch
More details here:
https://bitcointalk.org/index.php?topic=155140.0
Downgrading to 0.7.2 again, please help permanently fix this in the next 0.8.x release.
The text was updated successfully, but these errors were encountered: