Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upNew block database files platform-dependent? #2293
Comments
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
Did you copy the files while the application was running? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Belkaar
Feb 10, 2013
No, I shut it down and it was running with --detach-db=1 (if that matters)
Also: I tried this twice.
Belkaar
commented
Feb 10, 2013
|
No, I shut it down and it was running with --detach-db=1 (if that matters) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sipa
Feb 10, 2013
Member
Ok, interesting. Can you try:
- If you're able to compile on/for the RPi, can you try running the unit tests on it? (cd src; make -f makefile.unix test_bitcoin && ./test_bitcoin).
- Are you able to sync (even if it's very slow) on the RPi directly?
- Can you copy the files from an RPi to a desktop system, and continue the sync?
|
Ok, interesting. Can you try:
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Belkaar
Feb 10, 2013
- Will do
- It syncs and runs fine on the rpi and on the x86 machine. You just can't copy the db from one to the other (both ways will give errors)
- See 2
Belkaar
commented
Feb 10, 2013
|
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
Belkaar
commented
Feb 13, 2013
|
The test build said "Running 89 test cases, no errors" |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
rebroad
Mar 20, 2013
Contributor
AFAIK the database files on linux and MS windows are also incompatible with each other. Are there any plans to make it so that the files can be used for the client irrespective of platform/architecture?
|
AFAIK the database files on linux and MS windows are also incompatible with each other. Are there any plans to make it so that the files can be used for the client irrespective of platform/architecture? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
jgarzik
Mar 20, 2013
Contributor
This is low priority, given that the database may be regenerated from publicly available data at any time.
|
This is low priority, given that the database may be regenerated from publicly available data at any time. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Mar 8, 2014
Member
I just checked with master on my cubox-i (4 core, armv7l-linux-gnueabihf, debian jessie). Tried two scenarios:
- On ARM box
- bootstrap the first ? blocks using blk00000.dat file from other install
- shut down bitcoind
- copy resulting bitcoin directory to AMD64 box
- run bitcoind on AMD64 box
- On AMD64 box
- bootstrap the first 142647 blocks using bootstrap.dat file
- shut down bitcoind
- copy resulting bitcoin directory to ARM box
- run bitcoind on ARM box
Curiously, (1) seemed to work, whereas (2) manages to get past the leveldb validation but then starts to behave very erratic. It accepts blocks but never does anything with them, ie
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
....
Starting with -checkblocks=0 (which should check the entire database) gives
2014-03-08 07:47:38 Verifying last 142647 blocks at level 3
2014-03-08 07:51:22 No coin database inconsistencies in last 4083 blocks (216726 transactions)
2014-03-08 07:51:23 block index 233301ms
After a -reindex it works fine.
Edit: re-trying with the copied block index and -checkblocks=0 -checklevel=3 gives the same kind of errors OP has:
2014-03-08 08:21:54 Verifying last 142647 blocks at level 3
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
....
The daemon hangs after this.
|
I just checked with master on my cubox-i (4 core, armv7l-linux-gnueabihf, debian jessie). Tried two scenarios:
Curiously, (1) seemed to work, whereas (2) manages to get past the leveldb validation but then starts to behave very erratic. It accepts blocks but never does anything with them, ie Starting with -checkblocks=0 (which should check the entire database) gives After a -reindex it works fine. Edit: re-trying with the copied block index and The daemon hangs after this. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Mar 8, 2014
Member
OK this is curious. It it non-deterministic. Sometimes the verify passes, as pindexState gets 'stuck'
2014-03-08 08:45:20 Verifying pindexState=138564 pIndex=1
2014-03-08 08:45:20 No coin database inconsistencies in last 4083 blocks (216726 transactions)
2014-03-08 08:45:21 block index 244825ms
(this doesn't imply that the resulting index actually works, it doesn't)
|
OK this is curious. It it non-deterministic. Sometimes the verify passes, as pindexState gets 'stuck' (this doesn't imply that the resulting index actually works, it doesn't) |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
whitslack
Mar 24, 2014
I too tried to initialize a block index and chainstate database on my x86 system and then copy the files over to a Raspberry Pi. Since both systems are Linux, 32-bit, little-endian, I figured there would be no differences in the on-disk format. However, this is apparently not so. When Bitcoind starts verifying blocks, it immediately starts complaining about missing transactions. However, using the exact same files on the x86 machine (i.e., everything inside blocks/ and chainstate/ identical to what it was when starting on the RPi), Bitcoind starts up and verifies the blocks just fine.
Very disappointing that the on-disk format of these LevelDB databases differs between two architectures that are both 32-bit, little-endian.
whitslack
commented
Mar 24, 2014
|
I too tried to initialize a block index and chainstate database on my x86 system and then copy the files over to a Raspberry Pi. Since both systems are Linux, 32-bit, little-endian, I figured there would be no differences in the on-disk format. However, this is apparently not so. When Bitcoind starts verifying blocks, it immediately starts complaining about missing transactions. However, using the exact same files on the x86 machine (i.e., everything inside blocks/ and chainstate/ identical to what it was when starting on the RPi), Bitcoind starts up and verifies the blocks just fine. Very disappointing that the on-disk format of these LevelDB databases differs between two architectures that are both 32-bit, little-endian. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sipa
Mar 24, 2014
Member
I think this is unintentional, as the LevelDB on-disk format is very well described (up to the byte level). My guess is a LevelDB bug that only occurs on ARM.
We should probably report this upstream.
|
I think this is unintentional, as the LevelDB on-disk format is very well described (up to the byte level). My guess is a LevelDB bug that only occurs on ARM. We should probably report this upstream. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
whitslack
Mar 24, 2014
I guess we would need a minimal test case that produces different data files on ARM than on x86.
whitslack
commented
Mar 24, 2014
|
I guess we would need a minimal test case that produces different data files on ARM than on x86. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sipa
Mar 24, 2014
Member
It may just be a bug in Bitcoin's LevelDB glue code or serialization code too...
|
It may just be a bug in Bitcoin's LevelDB glue code or serialization code too... |
laanwj
added
the
UTXO Db and Indexes
label
May 6, 2014
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
May 7, 2014
Member
This is really strange. I've indexed the first four block files both on ARM and AMD64.
To compare the resulting leveldb files I've built a py-leveldb against bitcoin's leveldb and started poking at the database files, comparing them in various ways.
# Compare two bitcoind's leveldbs
import leveldb
dbdira = 'armtest'
dbdirb = 'amd64test'
databases = ['/chainstate', '/blocks/index']
def compare_databases(cs1, cs2):
ai = cs1.RangeIter()
bi = cs2.RangeIter()
n = 0
err = 0
aend = bend = False
while True:
try:
a = ai.next()
except StopIteration:
aend = True
try:
b = bi.next()
except StopIteration:
bend = True
if aend and bend:
break # both at end of iteration
if aend or bend or a != b:
err += 1
n += 1
print "Compared %i records" % n
print "Found %i differences" % err
def compare_databases2(cs1, cs2):
'''
Check that all keys of cs1 are in cs2
and that the values match.
'''
ai = cs1.RangeIter()
n = 0
err = 0
for (key,value) in ai:
try:
if cs2.Get(key) != value:
err += 1
except KeyError:
err += 1
n += 1
print "Compared %i records" % n
print "Found %i differences" % err
for db in databases:
print "* Comparing %s" % db
cs1 = leveldb.LevelDB(dbdira + db)
cs2 = leveldb.LevelDB(dbdirb + db)
print "** Check 1"
compare_databases(cs1, cs2)
print "** Check 2"
compare_databases2(cs1, cs2)
print "** Check 3"
compare_databases2(cs2, cs1)
printOutput:
* Comparing /chainstate
** Check 1
Compared 390331 records
Found 0 differences
** Check 2
Compared 390331 records
Found 0 differences
** Check 3
Compared 390331 records
Found 0 differences
* Comparing /blocks/index
** Check 1
Compared 142654 records
Found 0 differences
** Check 2
Compared 142654 records
Found 0 differences
** Check 3
Compared 142654 records
Found 0 differences
Seemingly, there are no differences at all in the data. I've also checked the blk* and rev*.dat files. They match to the byte. However, bitcoin on AMD64 refuses to work with the ARM files. I'm baffled.
Of course, bitcoin may have some other access pattern that brings to light a bug/difference in a lower level, but this is turning out to be elusive.
Also: Re-silvering the ARM-created databases by creating a new database and simply copying all keys/values, makes a database that can be used on AMD64.
|
This is really strange. I've indexed the first four block files both on ARM and AMD64. # Compare two bitcoind's leveldbs
import leveldb
dbdira = 'armtest'
dbdirb = 'amd64test'
databases = ['/chainstate', '/blocks/index']
def compare_databases(cs1, cs2):
ai = cs1.RangeIter()
bi = cs2.RangeIter()
n = 0
err = 0
aend = bend = False
while True:
try:
a = ai.next()
except StopIteration:
aend = True
try:
b = bi.next()
except StopIteration:
bend = True
if aend and bend:
break # both at end of iteration
if aend or bend or a != b:
err += 1
n += 1
print "Compared %i records" % n
print "Found %i differences" % err
def compare_databases2(cs1, cs2):
'''
Check that all keys of cs1 are in cs2
and that the values match.
'''
ai = cs1.RangeIter()
n = 0
err = 0
for (key,value) in ai:
try:
if cs2.Get(key) != value:
err += 1
except KeyError:
err += 1
n += 1
print "Compared %i records" % n
print "Found %i differences" % err
for db in databases:
print "* Comparing %s" % db
cs1 = leveldb.LevelDB(dbdira + db)
cs2 = leveldb.LevelDB(dbdirb + db)
print "** Check 1"
compare_databases(cs1, cs2)
print "** Check 2"
compare_databases2(cs1, cs2)
print "** Check 3"
compare_databases2(cs2, cs1)
printOutput: Seemingly, there are no differences at all in the data. I've also checked the blk* and rev*.dat files. They match to the byte. However, bitcoin on AMD64 refuses to work with the ARM files. I'm baffled. Of course, bitcoin may have some other access pattern that brings to light a bug/difference in a lower level, but this is turning out to be elusive. Also: Re-silvering the ARM-created databases by creating a new database and simply copying all keys/values, makes a database that can be used on AMD64. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
referenced this issue
Jun 4, 2014
Closed
LevelDB "Corruption: bad entry in block" on 0.9.1-beta #4263
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
OK, going to try |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Jun 5, 2014
Member
I tried using the leveldb files from ARM on AMD64 and vice-versa. Same problem as before - alas, the paranoid checks do not catch this.
|
I tried using the leveldb files from ARM on AMD64 and vice-versa. Same problem as before - alas, the paranoid checks do not catch this. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
sipa
Jun 5, 2014
Member
Thanks for testing!
I wonder: do we get any kind of warning/error in chainstate/LOG ?
|
Thanks for testing! I wonder: do we get any kind of warning/error in chainstate/LOG ? |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Jun 6, 2014
Member
I'll check next time.
I'm working on recording a trace of bitcoind's access pattern, then replaying that against the two database instance and seeing where returned values start to diverge.
|
I'll check next time. I'm working on recording a trace of bitcoind's access pattern, then replaying that against the two database instance and seeing where returned values start to diverge. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Jun 6, 2014
Member
I found the bug! Well at least I've narrowed it down a lot. The problem is in the filter_policy, which is set to leveldb::NewBloomFilterPolicy(10). Seemingly this filter policy data is not portable.
Commenting out that line makes the same block index files work on ARM and AMD64.
This explains why the database works fine after resilvering in my above post (and in general in py_leveldb, which doesn't use a filter policy...).
|
I found the bug! Well at least I've narrowed it down a lot. The problem is in the This explains why the database works fine after resilvering in my above post (and in general in py_leveldb, which doesn't use a filter policy...). |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Jun 7, 2014
Member
The Hash function used for the bloom filter was returning different values based on character signedness in some edge cases. See bitcoin-core/leveldb-old#5 for fix.
Upstream issue: https://code.google.com/p/leveldb/issues/detail?id=237
|
The Hash function used for the bloom filter was returning different values based on character signedness in some edge cases. See bitcoin-core/leveldb-old#5 for fix. |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
laanwj
Jul 1, 2014
Member
It doesn't seem that the upstream issue is being picked up. Platform compatibility of the databases is likely very low-priority for them.
We could work around this completely on the bitcoind side by providing our own FilterPolicy() implementation and passing it in with the options. That's just a small class that hashes keys and tests against them: https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/bloom.cc#L17 .
|
It doesn't seem that the upstream issue is being picked up. Platform compatibility of the databases is likely very low-priority for them. We could work around this completely on the bitcoind side by providing our own FilterPolicy() implementation and passing it in with the options. That's just a small class that hashes keys and tests against them: https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/bloom.cc#L17 . |
This comment has been minimized.
Show comment
Hide comment
This comment has been minimized.
|
See #5093 |
Belkaar commentedFeb 10, 2013
I compiled the content of the git tag 0.8-rc1 on my raspberry pi (ARM) and downloaded the blockchain (well most of it) into the new format.
When I try to copy the contents of blocks and chainstate an x86 machine and start the precompiled 0.8-rc1 bitcoind it crashes. Same happens with pi-created block chain files if you try to start the 0.8-rc1 windows.
(Copying the datafiles between x86 and ARM was no problem with 0.7)
Bitcoin version v0.8.0rc1-beta (2013-02-06 16:06:43 -0500)
Using OpenSSL version OpenSSL 0.9.8k 25 Mar 2009
Startup time: 2013-02-10 14:07:00
Default data directory /root/.bitcoin
Used data directory /mnt/store_0/bitcoin/bitcoincodes
init message: Verifying wallet integrity...
dbenv.open LogDir=/mnt/store_0/bitcoin/bitcoincodes/database ErrorFile=/mnt/store_0/bitcoin/bitcoincodes/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/blocks/index
Opened LevelDB successfully
Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/chainstate
Opened LevelDB successfully
LoadBlockIndex(): last block file = 37
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=133, size=23791072, heights=216794..216926, time=2013-01-16..2013-01-17)
LoadBlockIndex(): transaction index disabled
LoadBlockIndex(): hashBestChain=00000000000004e3fa816b4c834a13e581e446c2b6881c83b28f9864d615a663 height=216926 date=2013-01-17 19:45:03
init message: Verifying block database integrity...
Verifying last 288 blocks at level 3
ERROR: DisconnectBlock() : outputs still spent? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : outputs still spent? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
Unable to open file /mnt/store_0/bitcoin/bitcoincodes/blocks/rev00037.dat
ERROR: CBlockUndo::ReadFromDisk() : OpenBlockFile failed
ERROR: VerifyDB() : *** found bad undo data at 216819, hash=0000000000000333189f2bc4a671ed10abedb5c9fb3f6d69a2498c8c2317eeaa
Error: Corrupted block database detected. Please restart the client with -reindex.
Flush(false)
DBFlush(false) ended 0ms
StopNode()
Flushed 0 addresses to peers.dat 467ms
Committing 27465 changed transactions to coin database...
Flush(true)
DBFlush(true) ended 0ms
Bitcoin exited