New block database files platform-dependent? #2293

Belkaar · 2013-02-10T17:38:24Z

I compiled the content of the git tag 0.8-rc1 on my raspberry pi (ARM) and downloaded the blockchain (well most of it) into the new format.
When I try to copy the contents of blocks and chainstate an x86 machine and start the precompiled 0.8-rc1 bitcoind it crashes. Same happens with pi-created block chain files if you try to start the 0.8-rc1 windows.
(Copying the datafiles between x86 and ARM was no problem with 0.7)

Bitcoin version v0.8.0rc1-beta (2013-02-06 16:06:43 -0500)
Using OpenSSL version OpenSSL 0.9.8k 25 Mar 2009
Startup time: 2013-02-10 14:07:00
Default data directory /root/.bitcoin
Used data directory /mnt/store_0/bitcoin/bitcoincodes
init message: Verifying wallet integrity...
dbenv.open LogDir=/mnt/store_0/bitcoin/bitcoincodes/database ErrorFile=/mnt/store_0/bitcoin/bitcoincodes/db.log
Bound to [::]:8333
Bound to 0.0.0.0:8333
init message: Loading block index...
Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/blocks/index
Opened LevelDB successfully
Opening LevelDB in /mnt/store_0/bitcoin/bitcoincodes/chainstate
Opened LevelDB successfully
LoadBlockIndex(): last block file = 37
LoadBlockIndex(): last block file: CBlockFileInfo(blocks=133, size=23791072, heights=216794..216926, time=2013-01-16..2013-01-17)
LoadBlockIndex(): transaction index disabled
LoadBlockIndex(): hashBestChain=00000000000004e3fa816b4c834a13e581e446c2b6881c83b28f9864d615a663 height=216926 date=2013-01-17 19:45:03
init message: Verifying block database integrity...
Verifying last 288 blocks at level 3
ERROR: DisconnectBlock() : outputs still spent? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : outputs still spent? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : undo data adding output to missing transaction
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
ERROR: DisconnectBlock() : added transaction mismatch? database corrupted
Unable to open file /mnt/store_0/bitcoin/bitcoincodes/blocks/rev00037.dat
ERROR: CBlockUndo::ReadFromDisk() : OpenBlockFile failed
ERROR: VerifyDB() : *** found bad undo data at 216819, hash=0000000000000333189f2bc4a671ed10abedb5c9fb3f6d69a2498c8c2317eeaa

Error: Corrupted block database detected. Please restart the client with -reindex.
Flush(false)
DBFlush(false) ended 0ms
StopNode()
Flushed 0 addresses to peers.dat 467ms
Committing 27465 changed transactions to coin database...
Flush(true)
DBFlush(true) ended 0ms
Bitcoin exited

sipa · 2013-02-10T17:55:07Z

Did you copy the files while the application was running?

Belkaar · 2013-02-10T17:59:19Z

No, I shut it down and it was running with --detach-db=1 (if that matters)
Also: I tried this twice.

sipa · 2013-02-10T18:06:31Z

Ok, interesting. Can you try:

If you're able to compile on/for the RPi, can you try running the unit tests on it? (cd src; make -f makefile.unix test_bitcoin && ./test_bitcoin).
Are you able to sync (even if it's very slow) on the RPi directly?
Can you copy the files from an RPi to a desktop system, and continue the sync?

Belkaar · 2013-02-10T18:15:50Z

Will do
It syncs and runs fine on the rpi and on the x86 machine. You just can't copy the db from one to the other (both ways will give errors)
See 2

Belkaar · 2013-02-13T06:40:38Z

The test build said "Running 89 test cases, no errors"

rebroad · 2013-03-20T03:36:17Z

AFAIK the database files on linux and MS windows are also incompatible with each other. Are there any plans to make it so that the files can be used for the client irrespective of platform/architecture?

jgarzik · 2013-03-20T14:27:54Z

This is low priority, given that the database may be regenerated from publicly available data at any time.

laanwj · 2014-03-08T08:07:37Z

I just checked with master on my cubox-i (4 core, armv7l-linux-gnueabihf, debian jessie). Tried two scenarios:

On ARM box

bootstrap the first ? blocks using blk00000.dat file from other install
shut down bitcoind
copy resulting bitcoin directory to AMD64 box
run bitcoind on AMD64 box

On AMD64 box

bootstrap the first 142647 blocks using bootstrap.dat file
shut down bitcoind
copy resulting bitcoin directory to ARM box
run bitcoind on ARM box

Curiously, (1) seemed to work, whereas (2) manages to get past the leveldb validation but then starts to behave very erratic. It accepts blocks but never does anything with them, ie

2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
2014-03-08 07:51:31 ProcessBlock: ACCEPTED
....

Starting with -checkblocks=0 (which should check the entire database) gives

2014-03-08 07:47:38 Verifying last 142647 blocks at level 3
2014-03-08 07:51:22 No coin database inconsistencies in last 4083 blocks (216726 transactions)
2014-03-08 07:51:23  block index          233301ms

After a -reindex it works fine.

Edit: re-trying with the copied block index and -checkblocks=0 -checklevel=3 gives the same kind of errors OP has:

2014-03-08 08:21:54 Verifying last 142647 blocks at level 3
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
2014-03-08 08:22:57 ERROR: DisconnectBlock() : undo data adding output to missing transaction
....

The daemon hangs after this.

laanwj · 2014-03-08T08:48:56Z

OK this is curious. It it non-deterministic. Sometimes the verify passes, as pindexState gets 'stuck'

2014-03-08 08:45:20 Verifying pindexState=138564 pIndex=1
2014-03-08 08:45:20 No coin database inconsistencies in last 4083 blocks (216726 transactions)
2014-03-08 08:45:21  block index          244825ms

(this doesn't imply that the resulting index actually works, it doesn't)

whitslack · 2014-03-24T12:20:41Z

I too tried to initialize a block index and chainstate database on my x86 system and then copy the files over to a Raspberry Pi. Since both systems are Linux, 32-bit, little-endian, I figured there would be no differences in the on-disk format. However, this is apparently not so. When Bitcoind starts verifying blocks, it immediately starts complaining about missing transactions. However, using the exact same files on the x86 machine (i.e., everything inside blocks/ and chainstate/ identical to what it was when starting on the RPi), Bitcoind starts up and verifies the blocks just fine.

Very disappointing that the on-disk format of these LevelDB databases differs between two architectures that are both 32-bit, little-endian.

sipa · 2014-03-24T12:29:46Z

I think this is unintentional, as the LevelDB on-disk format is very well described (up to the byte level). My guess is a LevelDB bug that only occurs on ARM.

We should probably report this upstream.

whitslack · 2014-03-24T12:33:00Z

I guess we would need a minimal test case that produces different data files on ARM than on x86.

sipa · 2014-03-24T12:36:09Z

It may just be a bug in Bitcoin's LevelDB glue code or serialization code too...

laanwj · 2014-05-07T19:03:40Z

This is really strange. I've indexed the first four block files both on ARM and AMD64.
To compare the resulting leveldb files I've built a py-leveldb against bitcoin's leveldb and started poking at the database files, comparing them in various ways.

# Compare two bitcoind's leveldbs
import leveldb

dbdira = 'armtest'
dbdirb = 'amd64test'

databases = ['/chainstate', '/blocks/index']

def compare_databases(cs1, cs2):
    ai = cs1.RangeIter()
    bi = cs2.RangeIter()

    n = 0
    err = 0
    aend = bend = False
    while True:
        try:
            a = ai.next()
        except StopIteration:
            aend = True
        try:
            b = bi.next()
        except StopIteration:
            bend = True
        if aend and bend:
            break # both at end of iteration
        if aend or bend or a != b:
            err += 1
        n += 1

    print "Compared %i records" % n
    print "Found %i differences" % err

def compare_databases2(cs1, cs2):
    '''
    Check that all keys of cs1 are in cs2
    and that the values match.
    '''
    ai = cs1.RangeIter()

    n = 0
    err = 0
    for (key,value) in ai:
        try:
            if cs2.Get(key) != value:
                err += 1
        except KeyError:
            err += 1
        n += 1

    print "Compared %i records" % n
    print "Found %i differences" % err

for db in databases:
    print "* Comparing %s" % db
    cs1 = leveldb.LevelDB(dbdira + db)
    cs2 = leveldb.LevelDB(dbdirb + db)
    print "** Check 1"
    compare_databases(cs1, cs2)
    print "** Check 2"
    compare_databases2(cs1, cs2)
    print "** Check 3"
    compare_databases2(cs2, cs1)
    print

Output:

* Comparing /chainstate
** Check 1
Compared 390331 records
Found 0 differences
** Check 2
Compared 390331 records
Found 0 differences
** Check 3
Compared 390331 records
Found 0 differences

* Comparing /blocks/index
** Check 1
Compared 142654 records
Found 0 differences
** Check 2
Compared 142654 records
Found 0 differences
** Check 3
Compared 142654 records
Found 0 differences

Seemingly, there are no differences at all in the data. I've also checked the blk* and rev*.dat files. They match to the byte. However, bitcoin on AMD64 refuses to work with the ARM files. I'm baffled.

Of course, bitcoin may have some other access pattern that brings to light a bug/difference in a lower level, but this is turning out to be elusive.

Also: Re-silvering the ARM-created databases by creating a new database and simply copying all keys/values, makes a database that can be used on AMD64.

sipa · 2014-05-31T15:21:31Z

@laanwj Feel like trying again with #4161 and #4177 merged?

laanwj · 2014-06-05T07:08:42Z

OK, going to try

laanwj · 2014-06-05T11:54:11Z

I tried using the leveldb files from ARM on AMD64 and vice-versa. Same problem as before - alas, the paranoid checks do not catch this.

sipa · 2014-06-05T23:45:35Z

Thanks for testing!

I wonder: do we get any kind of warning/error in chainstate/LOG ?

laanwj · 2014-06-06T14:54:08Z

I'll check next time.

I'm working on recording a trace of bitcoind's access pattern, then replaying that against the two database instance and seeing where returned values start to diverge.

laanwj · 2014-06-06T16:42:42Z

I found the bug! Well at least I've narrowed it down a lot. The problem is in the filter_policy, which is set to leveldb::NewBloomFilterPolicy(10). Seemingly this filter policy data is not portable.
Commenting out that line makes the same block index files work on ARM and AMD64.

This explains why the database works fine after resilvering in my above post (and in general in py_leveldb, which doesn't use a filter policy...).

laanwj · 2014-06-07T11:36:11Z

The Hash function used for the bloom filter was returning different values based on character signedness in some edge cases. See bitcoin-core/leveldb-old#5 for fix.
Upstream issue: https://code.google.com/p/leveldb/issues/detail?id=237

laanwj · 2014-07-01T05:20:31Z

It doesn't seem that the upstream issue is being picked up. Platform compatibility of the databases is likely very low-priority for them.

We could work around this completely on the bitcoind side by providing our own FilterPolicy() implementation and passing it in with the options. That's just a small class that hashes keys and tests against them: https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/bloom.cc#L17 .

laanwj · 2014-10-21T07:54:43Z

See #5093

laanwj added the UTXO Db and Indexes label May 6, 2014

laanwj mentioned this issue Jun 4, 2014

LevelDB "Corruption: bad entry in block" on 0.9.1-beta #4263

Closed

laanwj mentioned this issue Jun 7, 2014

Make Hash independent of char signedness bitcoin-core/leveldb-old#5

Closed

laanwj mentioned this issue Oct 21, 2014

Upgrade to LevelDB 1.18 #5093

Merged

laanwj closed this as completed Oct 27, 2014

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

New block database files platform-dependent? #2293

New block database files platform-dependent? #2293

Belkaar commented Feb 10, 2013

sipa commented Feb 10, 2013

Belkaar commented Feb 10, 2013

sipa commented Feb 10, 2013

Belkaar commented Feb 10, 2013

Belkaar commented Feb 13, 2013

rebroad commented Mar 20, 2013

jgarzik commented Mar 20, 2013

laanwj commented Mar 8, 2014

laanwj commented Mar 8, 2014

whitslack commented Mar 24, 2014

sipa commented Mar 24, 2014

whitslack commented Mar 24, 2014

sipa commented Mar 24, 2014

laanwj commented May 7, 2014

sipa commented May 31, 2014

laanwj commented Jun 5, 2014

laanwj commented Jun 5, 2014

sipa commented Jun 5, 2014

laanwj commented Jun 6, 2014

laanwj commented Jun 6, 2014

laanwj commented Jun 7, 2014

laanwj commented Jul 1, 2014

laanwj commented Oct 21, 2014

New block database files platform-dependent? #2293

New block database files platform-dependent? #2293

Comments

Belkaar commented Feb 10, 2013

sipa commented Feb 10, 2013

Belkaar commented Feb 10, 2013

sipa commented Feb 10, 2013

Belkaar commented Feb 10, 2013

Belkaar commented Feb 13, 2013

rebroad commented Mar 20, 2013

jgarzik commented Mar 20, 2013

laanwj commented Mar 8, 2014

laanwj commented Mar 8, 2014

whitslack commented Mar 24, 2014

sipa commented Mar 24, 2014

whitslack commented Mar 24, 2014

sipa commented Mar 24, 2014

laanwj commented May 7, 2014

sipa commented May 31, 2014

laanwj commented Jun 5, 2014

laanwj commented Jun 5, 2014

sipa commented Jun 5, 2014

laanwj commented Jun 6, 2014

laanwj commented Jun 6, 2014

laanwj commented Jun 7, 2014

laanwj commented Jul 1, 2014

laanwj commented Oct 21, 2014