Cache transaction validation successes

[sipa]

This is an alternative to @gavinandresen's #5835, which caches transaction validation results in an mru cache in main, rather than relying on the mempool in consensus code. The cache keeps the last 33k results, and uses around 3 MB on 64-bit systems.

This pull request also contains a rebased version of the unit test from #5835, and the optimizations from #6064.

[sipa]

Reimplemented the mrumap using boost::multiindex, which is more compact, and offers erasing elements too. Entries are now removed from the tx validation cache when they are accepted into a block, meaning the last 33k valid unconfirmed transactions are cached, rather than just the last 33k whatsoever.

[sipa]

Rebased on new #6064.

[gavinandresen]

Concept ACK. I hope to run some benchmarks to see how much this speeds up validation with a stream of real 'tx' and 'block' messages and different -dbcache settings, but that probably won't happen until I'm back from New York late next week.

I also want to benchmark #5967 and #6102 .

[sipa]

Rebased.

[sipa]

Actually rebased.

[sipa]

Been running on bitcoin.sipa.be for a week, no problems.

[sipa]

ping

[jgarzik]

"it works" positive & negative test + review ACK

[sipa]

Rebased.

[laanwj]

ACK

[laanwj]

Backported to 0.11 as bc484ef

[jtimon]

I missed this..
We want to cache unconfirmed transactions, sure. But why cache the transactions that are being confirmed in connectBlock?

EDIT: As in, couldn't this have been done in AcceptToMemoryPool instead of CheckScript (which, by the way, it's called twice inside AcceptToMemoryPool)?

[sipa]

@jtimon To avoid rechecking the same transaction when it enters a block, when it has already been verified for the mempool (or whatever other reason).

[jtimon]

@sipa Thanks, I guess this the simplest way to do that right now.

[sdaftuar]

As mentioned on IRC, I think there's a problem with transaction checks involving block height. Consider a coinbase spend that is valid when the transaction arrives and is accepted into the mempool, but is invalid at any earlier block height because of the coinbase maturity test. If there's a reorg, then when validating blocks at earlier heights, we'd consider blocks valid even if they include this transaction at a too-early height, because of this validation cache. (A test that demonstrates this is here: https://gist.github.com/sdaftuar/23a3db523e68541a3195)

[jtimon]

@sdaftuar now that you say it, that's why I repeated the checks even when the transaction had been fully validated here: jtimon@935ee1e#diff-7ec3c68a81efff79b6ca22ac1f1eabbaR779
Maybe I was too fast closing #6445 (which should solve the problem) after all...

[sipa]

Going to revert, thanks for pointing this out, @sdaftuar.

[jtimon]

This wasn't fully reverted, sipa@517e6dd broke #6235 and #6382
@laanwj don't merge #6235 because that would break the build on master. Fixing now...