Add optional transaction index to databases

[sipa]

By specifying -txindex when initializing the database, a txid-to-diskpos index is maintained in the blktree database. This database is used to help answering getrawtransaction() RPC queries, when enabled.

Changing the -txindex value requires a -reindex; the client will abort at startup if the database and the specified -txindex mismatch.

[Diapolo]

This looks a little weird, can you put that } at the end of the above line?

[Diapolo]

How much will this increase a nodes load / how much larger will the resulting database be?

[sipa]

@Diapolo Around 500 MB extra in storage, and a lot of extra I/O. I didn't spend much effort optimizing this, as I don't consider this functionality a priority.

[Diapolo]

@sipa Thanks, I was just interested in the technical base aspects :).

[mikehearn]

Could you grab a service bit and make "getdata" use the new index too? Or is that too much additional work for this change?

[sipa]

@mikehearn I'm absolutely against making this available to the P2P network. If there is one thing I don't want services to depend on, then it is the availability of a fully indexed transaction history. If you really need one, fine, but maintain it yourself.

[BitcoinPullTester]

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/1f4691130ea0c54c69ceb04407507cd80dd05133 for binaries and test log.

[mikehearn]

I'm not sure I see the issue. Yes you can make a node do lots of disk IO, but downloading the block chain does that too. It'd be nice to fix, but it's not a new attack vector.

Apps that need indexes like that will end up just using blockchain.info or various random other sites/protocols to get what they want. It's not like those apps will go away if the P2P network doesn't give them what they need.

[sipa]

The only thing keeping history is necessary for, is bootstrapping fully validating nodes. However, that doesn't require an index (or even a Bitcoin node at all - it could be provided by an HTTP-based file service or other protocols - something that web has plenty of). I'm sure that making a feature available on the P2P will result in infrastructure depending on it, something that would burden the nodes that provide archive data. Yes, making it an optional feature makes this much less of a problem, but I still prefer to be conservative with functions the P2P network provides unless there is a very clear use.

As far as I know, all versions before 0.7.0 maintained a full tx-to-diskpos index, and there was no way at all to query it (not even RPC). I'm quite sure this was a deliberate choice by Satoshi. In a mail about maintaining per-txout spendability of wallet transactions, he clearly said not to rely on the txindex for this, as it wouldn't always be available.

EDIT: I dislike this feature as whole (even as RPC), as I think that services that are built not to depend on infinite history have better scalability. It's extremely useful for debugging though, so I added it. I'm sure people will use it for other purposes as well, and that's clearly preferable to having them depend on a centralized service for it. Providing it via P2P doesn't add much to that, IMHO.

[mikehearn]

OK, good points. Now I agree with you.

[BitcoinPullTester]

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/2d1fa42e85c9164688aa69b3f54f015fbefc06aa for binaries and test log.

[gavinandresen]

Test plan that I'll finish executing after lunch:

Default arguments, getrawtransaction w. arbitrary old, not-in-your-wallet txid, fully-spent txn
EXPECT: transaction not found

Rerun with -txindex=1
EXPECT: startup fails with "must -reindex" message

Note disk space used by blktree/ subdirectory
Rerun with -txindex=1 -reindex=1 -logtimestamps=1
EXPECT: long background process to rebuild index (Q: how long?)
Wait until "Reindexing finished" appears in debug.log, then:
getrawtransaction
EXPECT: success
Q: How much extra disk space ?

Rerun with -txindex=0:
EXPECT: startup fails with "must -reindex" message

Note disk space used by blktree/
Rerun with -txindex=0 -reindex=1 -logtimestamps=1
EXPECT: long background process to rebuild index (Q: how long?)
Wait until "Reindexing finished" appears in debug.log, then:
getrawtransaction w. old txid
EXPECT: transaction not found
EXPECT: txindex disk space freed from blktree/

[sipa]

@gavinandresen RE test plan:

• getrawtransaction without txindex should work for not-fully spent confirmed transactions, mempool transactions or transactions in the relay cache. For not-fully spent confirmed transactions, it may be slower than with txindex present. In general, without txindex I consider getrawtransaction to just work on a best-effort basis.
• extra disk space caused by txindex will be in the blktree/ directory

Otherwise the plan looks correct and complete to me.

EDIT: -reindex doesn't cause a slow startup, the startup is always instant, but importing (and thus building the txindex) happens in the background.

[gavinandresen]

Test results: success!

Disk space: blktree/ 30M --> 600M

Time to -reindex: (note: running -g build, no -dbcache set)
1 hour 50 minutes (same time, with/without -fullindex)

I'm going to pull.

[aaronash]

Just wanted to say thanks for working on this. Being able to getrawtransaction on pretty much anything removes reliance on 3rd party services, which I find very valuable.