Add -reindex, to perform in-place reindexing of blockchain data files

[jgarzik]

Hopefully this is done in a database-independent manner, with minimal interference to the upcoming ultraprune.

[BitcoinPullTester]

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/68e760d2656716cc9d538567c0735c77d8e5749e for binaries and test log.

[sipa]

I really wouldn't delete bdb database files yourself. Use the dbenv methods to do so.

[gavinandresen]

I agree with sipa, dbenv seemed to get very confused when I renamed database files behind its back for my corrupt wallet.dat handling code.

[sipa]

Surprising how little code changes are necessary for this. Ultraprune already has a CDiskBlockPos however, which is also used inside CTxBlockPos, so maybe you can use that instead? I think you can just copy it, and remove the IMPLEMENT_SERIALIZE.

[sipa]

I also wonder about this: what if the current best chain happens to be known in the new/repained blockchain? There should be no need to rebuild the entire index in that case, only the block positions.

EDIT: right, you can't, as in the current database structure, transactions also contain disk positions. It's something that can be done post-ultraprune though.

[jgarzik]

Updated with the following changes:

• uses CDiskBlockPos from sipa's ultraprune branch
• calls our CDBEnv to perform database removal

[sipa]

Have you tested this?

$ ./bitcoind -reindex
DB_ENV->dbremove: method not permitted before handle's open method

[gavinandresen]

I wrote up a 4-step test plan that I've been working through.

I also have a branch (not published) to reconcile this with my corrupt wallet handling pull; @jgarzik see https://github.com/gavinandresen/bitcoin-git/commits/BDB_RECOVER

The testplan I was working through:

1. Run new testnet3-in-a-box instance, let it IBD, shut it down. Remove testnet3/database/ directory.
   EXPECT: automatic re-index of blk0001.dat on startup.
2. Same a (1), but run with -detachdb. Corrupt blkindex.dat by truncating it by a few bytes, then restart.
   EXPECT: automatic re-index of blk0001.dat on startup.
3. Same as (2), but corrupt some bytes instead of truncating the file.
   EXPECT: automatic re-index.
4. Compile a bitcoind against a later version of BDB (e.g. BDB 5 instead of 4). Run without -detachdb.
   Then startup a bitcoind compiled against the earlier BDB version.
   EXPECT: Something Reasonable-- not a BDB_RUNRECOVERY uncaught exception/crash.
5. Same as (4), run with -detachdb.
   EXPECT: Same something reasonable as (4).

I haven't tested 4/5 yet, and don't know enough about BDB major version compatibility to know what is reasonable to expect.

[jgarzik]

Doing the database remove in the environment definitely creates a super-large removal transactions, yet does not actually remove blkindex.dat. blk0002.dat is growing here, which should not be the case. Further debugging is needed.

[gavinandresen]

Instead of: dbenv.dbremove(NULL, strFile.c_str(), "main", DB_AUTO_COMMIT); you might try dbenv.dbremove(NULL, strFile.c_str(), NULL, DB_AUTO_COMMIT);
... then re-create blkindex.dat using TxDb("cr")

I think passing NULL as the third argument removes the file.

[jgarzik]

Fixed a few bugs, and rebased on top of @gavinandresen 's ::RemoveDb(). blkindex.dat is now properly removed, and reindex proceeds as expected. Things appear to be working now.

Caveat: a genesis block is newly stored to blkNNNN.dat, for each -reindex invocation. LoadBlockIndex() adds a new one to the (it thinks) newly created block file. All other blocks are properly read and processed in-place.

-reindex highlights the poor speed of BDB indexing alone... and should provide a useful apples-to-apples basis for comparing the old BDB system with ultraprune.

[sipa]

Needs rebase.

[BitcoinPullTester]

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/f49f2631856a4a233a957ce8361ffab3b08561cc for binaries and test log.

[jgarzik]

Superceded by #1943, closing