New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New block database files platform-dependent? #2293
Comments
Did you copy the files while the application was running? |
No, I shut it down and it was running with --detach-db=1 (if that matters) |
Ok, interesting. Can you try:
|
|
The test build said "Running 89 test cases, no errors" |
AFAIK the database files on linux and MS windows are also incompatible with each other. Are there any plans to make it so that the files can be used for the client irrespective of platform/architecture? |
This is low priority, given that the database may be regenerated from publicly available data at any time. |
I just checked with master on my cubox-i (4 core, armv7l-linux-gnueabihf, debian jessie). Tried two scenarios:
Curiously, (1) seemed to work, whereas (2) manages to get past the leveldb validation but then starts to behave very erratic. It accepts blocks but never does anything with them, ie
Starting with -checkblocks=0 (which should check the entire database) gives
After a -reindex it works fine. Edit: re-trying with the copied block index and
The daemon hangs after this. |
OK this is curious. It it non-deterministic. Sometimes the verify passes, as pindexState gets 'stuck'
(this doesn't imply that the resulting index actually works, it doesn't) |
I too tried to initialize a block index and chainstate database on my x86 system and then copy the files over to a Raspberry Pi. Since both systems are Linux, 32-bit, little-endian, I figured there would be no differences in the on-disk format. However, this is apparently not so. When Bitcoind starts verifying blocks, it immediately starts complaining about missing transactions. However, using the exact same files on the x86 machine (i.e., everything inside blocks/ and chainstate/ identical to what it was when starting on the RPi), Bitcoind starts up and verifies the blocks just fine. Very disappointing that the on-disk format of these LevelDB databases differs between two architectures that are both 32-bit, little-endian. |
I think this is unintentional, as the LevelDB on-disk format is very well described (up to the byte level). My guess is a LevelDB bug that only occurs on ARM. We should probably report this upstream. |
I guess we would need a minimal test case that produces different data files on ARM than on x86. |
It may just be a bug in Bitcoin's LevelDB glue code or serialization code too... |
This is really strange. I've indexed the first four block files both on ARM and AMD64. # Compare two bitcoind's leveldbs
import leveldb
dbdira = 'armtest'
dbdirb = 'amd64test'
databases = ['/chainstate', '/blocks/index']
def compare_databases(cs1, cs2):
ai = cs1.RangeIter()
bi = cs2.RangeIter()
n = 0
err = 0
aend = bend = False
while True:
try:
a = ai.next()
except StopIteration:
aend = True
try:
b = bi.next()
except StopIteration:
bend = True
if aend and bend:
break # both at end of iteration
if aend or bend or a != b:
err += 1
n += 1
print "Compared %i records" % n
print "Found %i differences" % err
def compare_databases2(cs1, cs2):
'''
Check that all keys of cs1 are in cs2
and that the values match.
'''
ai = cs1.RangeIter()
n = 0
err = 0
for (key,value) in ai:
try:
if cs2.Get(key) != value:
err += 1
except KeyError:
err += 1
n += 1
print "Compared %i records" % n
print "Found %i differences" % err
for db in databases:
print "* Comparing %s" % db
cs1 = leveldb.LevelDB(dbdira + db)
cs2 = leveldb.LevelDB(dbdirb + db)
print "** Check 1"
compare_databases(cs1, cs2)
print "** Check 2"
compare_databases2(cs1, cs2)
print "** Check 3"
compare_databases2(cs2, cs1)
print Output:
Seemingly, there are no differences at all in the data. I've also checked the blk* and rev*.dat files. They match to the byte. However, bitcoin on AMD64 refuses to work with the ARM files. I'm baffled. Of course, bitcoin may have some other access pattern that brings to light a bug/difference in a lower level, but this is turning out to be elusive. Also: Re-silvering the ARM-created databases by creating a new database and simply copying all keys/values, makes a database that can be used on AMD64. |
OK, going to try |
I tried using the leveldb files from ARM on AMD64 and vice-versa. Same problem as before - alas, the paranoid checks do not catch this. |
Thanks for testing! I wonder: do we get any kind of warning/error in chainstate/LOG ? |
I'll check next time. I'm working on recording a trace of bitcoind's access pattern, then replaying that against the two database instance and seeing where returned values start to diverge. |
I found the bug! Well at least I've narrowed it down a lot. The problem is in the This explains why the database works fine after resilvering in my above post (and in general in py_leveldb, which doesn't use a filter policy...). |
The Hash function used for the bloom filter was returning different values based on character signedness in some edge cases. See bitcoin-core/leveldb-old#5 for fix. |
It doesn't seem that the upstream issue is being picked up. Platform compatibility of the databases is likely very low-priority for them. We could work around this completely on the bitcoind side by providing our own FilterPolicy() implementation and passing it in with the options. That's just a small class that hashes keys and tests against them: https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/bloom.cc#L17 . |
See #5093 |
I compiled the content of the git tag 0.8-rc1 on my raspberry pi (ARM) and downloaded the blockchain (well most of it) into the new format.
When I try to copy the contents of blocks and chainstate an x86 machine and start the precompiled 0.8-rc1 bitcoind it crashes. Same happens with pi-created block chain files if you try to start the 0.8-rc1 windows.
(Copying the datafiles between x86 and ARM was no problem with 0.7)
Bitcoin version v0.8.0rc1-beta (2013-02-06 16:06:43 -0500)
Using OpenSSL version OpenSSL 0.9.8k 25 Mar 2009
Startup time: 2013-02-10 14:07:00
Default data directory /root/.bitcoin
Used data directory /mnt/store_0/bitcoin/bitcoincodes
init message: Verifying wallet integrity...
dbenv.open LogDir=/mnt/store_0/bitcoin/bitcoincodes/database ErrorFile=/mnt/store_0/bitcoin/bitcoincodes/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/blocks/index
Opened LevelDB successfully
Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/chainstate
Opened LevelDB successfully
LoadBlockIndex(): last block file = 37
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=133, size=23791072, heights=216794..216926, time=2013-01-16..2013-01-17)
LoadBlockIndex(): transaction index disabled
LoadBlockIndex(): hashBestChain=00000000000004e3fa816b4c834a13e581e446c2b6881c83b28f9864d615a663 height=216926 date=2013-01-17 19:45:03
init message: Verifying block database integrity...
Verifying last 288 blocks at level 3
ERROR: DisconnectBlock() : outputs still spent? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : outputs still spent? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
Unable to open file /mnt/store_0/bitcoin/bitcoincodes/blocks/rev00037.dat
ERROR: CBlockUndo::ReadFromDisk() : OpenBlockFile failed
ERROR: VerifyDB() : *** found bad undo data at 216819, hash=0000000000000333189f2bc4a671ed10abedb5c9fb3f6d69a2498c8c2317eeaa
Error: Corrupted block database detected. Please restart the client with -reindex.
Flush(false)
DBFlush(false) ended 0ms
StopNode()
Flushed 0 addresses to peers.dat 467ms
Committing 27465 changed transactions to coin database...
Flush(true)
DBFlush(true) ended 0ms
Bitcoin exited
The text was updated successfully, but these errors were encountered: