Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache transaction validation successes #6077

Merged
merged 2 commits into from Jul 27, 2015
Merged

Cache transaction validation successes #6077

merged 2 commits into from Jul 27, 2015

Conversation

@sipa
Copy link
Member

sipa commented Apr 28, 2015

This is an alternative to @gavinandresen's #5835, which caches transaction validation results in an mru cache in main, rather than relying on the mempool in consensus code. The cache keeps the last 33k results, and uses around 3 MB on 64-bit systems.

This pull request also contains a rebased version of the unit test from #5835, and the optimizations from #6064.

@sipa sipa changed the title Cache transactionc validation successes Cache transaction validation successes Apr 28, 2015
@laanwj laanwj added the Validation label Apr 29, 2015
@sipa sipa force-pushed the sipa:cachetxcheck branch from 47a291e to 31f67c3 Apr 29, 2015
@sipa
Copy link
Member Author

sipa commented Apr 29, 2015

Reimplemented the mrumap using boost::multiindex, which is more compact, and offers erasing elements too. Entries are now removed from the tx validation cache when they are accepted into a block, meaning the last 33k valid unconfirmed transactions are cached, rather than just the last 33k whatsoever.

@sipa sipa force-pushed the sipa:cachetxcheck branch 3 times, most recently from 0dd16da to 3f29d91 Apr 29, 2015
@sipa
Copy link
Member Author

sipa commented Apr 30, 2015

Rebased on new #6064.

@sipa sipa force-pushed the sipa:cachetxcheck branch from 3f29d91 to 41fd063 May 1, 2015
@gavinandresen
Copy link
Contributor

gavinandresen commented May 15, 2015

Concept ACK. I hope to run some benchmarks to see how much this speeds up validation with a stream of real 'tx' and 'block' messages and different -dbcache settings, but that probably won't happen until I'm back from New York late next week.

I also want to benchmark #5967 and #6102 .

@sipa sipa force-pushed the sipa:cachetxcheck branch from 41fd063 Jun 14, 2015
@sipa
Copy link
Member Author

sipa commented Jun 14, 2015

Rebased.

@sipa sipa force-pushed the sipa:cachetxcheck branch Jun 16, 2015
@sipa
Copy link
Member Author

sipa commented Jun 16, 2015

Actually rebased.

@sipa
Copy link
Member Author

sipa commented Jun 21, 2015

Been running on bitcoin.sipa.be for a week, no problems.

@sipa
Copy link
Member Author

sipa commented Jul 27, 2015

@jgarzik
Copy link
Contributor

jgarzik commented Jul 27, 2015

"it works" positive & negative test + review ACK

@sipa sipa force-pushed the sipa:cachetxcheck branch Jul 27, 2015
@sipa
Copy link
Member Author

sipa commented Jul 27, 2015

Rebased.

As suggested by Greg Maxwell-- unit test to make sure a block
with a double-spend in it doesn't pass validation if half of
the double-spend is already in the memory pool (so full-blown
transaction validation is skipped) when the block is received.
@sipa sipa force-pushed the sipa:cachetxcheck branch to 517e6dd Jul 27, 2015
@laanwj
Copy link
Member

laanwj commented Jul 27, 2015

ACK

@laanwj laanwj merged commit 517e6dd into bitcoin:master Jul 27, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
laanwj added a commit that referenced this pull request Jul 27, 2015
517e6dd Unit test doublespends in new blocks (Gavin Andresen)
17b1142 Cache transaction validation successes (Pieter Wuille)
sipa added a commit that referenced this pull request Jul 27, 2015
Conflicts:
	src/main.cpp
	src/test/test_bitcoin.cpp

Github-Pull: #6077
Rebased-From: 17b1142 517e6dd
@laanwj
Copy link
Member

laanwj commented Jul 27, 2015

Backported to 0.11 as bc484ef

@jtimon
Copy link
Member

jtimon commented Jul 28, 2015

I missed this..
We want to cache unconfirmed transactions, sure. But why cache the transactions that are being confirmed in connectBlock?

EDIT: As in, couldn't this have been done in AcceptToMemoryPool instead of CheckScript (which, by the way, it's called twice inside AcceptToMemoryPool)?

@sipa
Copy link
Member Author

sipa commented Jul 28, 2015

@jtimon To avoid rechecking the same transaction when it enters a block, when it has already been verified for the mempool (or whatever other reason).

@jtimon
Copy link
Member

jtimon commented Jul 28, 2015

@sipa Thanks, I guess this the simplest way to do that right now.

@sdaftuar
Copy link
Member

sdaftuar commented Jul 28, 2015

As mentioned on IRC, I think there's a problem with transaction checks involving block height. Consider a coinbase spend that is valid when the transaction arrives and is accepted into the mempool, but is invalid at any earlier block height because of the coinbase maturity test. If there's a reorg, then when validating blocks at earlier heights, we'd consider blocks valid even if they include this transaction at a too-early height, because of this validation cache. (A test that demonstrates this is here: https://gist.github.com/sdaftuar/23a3db523e68541a3195)

@jtimon
Copy link
Member

jtimon commented Jul 28, 2015

@sdaftuar now that you say it, that's why I repeated the checks even when the transaction had been fully validated here: jtimon@935ee1e#diff-7ec3c68a81efff79b6ca22ac1f1eabbaR779
Maybe I was too fast closing #6445 (which should solve the problem) after all...

@sipa
Copy link
Member Author

sipa commented Jul 28, 2015

Going to revert, thanks for pointing this out, @sdaftuar.

@jtimon
Copy link
Member

jtimon commented Aug 21, 2015

This wasn't fully reverted, sipa@517e6dd broke #6235 and #6382
@laanwj don't merge #6235 because that would break the build on master. Fixing now...

laanwj added a commit that referenced this pull request Oct 9, 2015
Reported by sipa
zkbot added a commit to zcash/zcash that referenced this pull request Mar 21, 2018
Misc upstream PRs

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6077
  - Second commit only (first was already applied to 0.11.X and then reverted)
- bitcoin/bitcoin#6284
- bitcoin/bitcoin#6489
- bitcoin/bitcoin#6462
- bitcoin/bitcoin#6647
- bitcoin/bitcoin#6235
- bitcoin/bitcoin#6905
- bitcoin/bitcoin#6780
  - Excluding second commit (QT) and third commit (requires bitcoin/bitcoin#6993)
- bitcoin/bitcoin#6961
  - Excluding QT parts, and a small `src/policy/policy.cpp` change which depends on a bunch of other PRs, which we'll have to remember to come back to.
- bitcoin/bitcoin#7044
- bitcoin/bitcoin#8856
- bitcoin/bitcoin#9002

Part of #2074 and #2132.
zkbot added a commit to zcash/zcash that referenced this pull request Dec 4, 2019
Misc upstream PRs

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6077
  - Second commit only (first was already applied to 0.11.X and then reverted)
- bitcoin/bitcoin#6284
- bitcoin/bitcoin#6489
- bitcoin/bitcoin#6235
- bitcoin/bitcoin#6905
- bitcoin/bitcoin#6780
  - Excluding second commit (QT) and third commit (requires bitcoin/bitcoin#6993)
- bitcoin/bitcoin#6961
  - Excluding QT parts, and a small `src/policy/policy.cpp` change which depends on a bunch of other PRs, which we'll have to remember to come back to.
- bitcoin/bitcoin#7044
- bitcoin/bitcoin#8856
- bitcoin/bitcoin#9002

Part of #2074 and #2132.
zkbot added a commit to zcash/zcash that referenced this pull request Dec 4, 2019
Misc upstream PRs

Cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6077
  - Second commit only (first was already applied to 0.11.X and then reverted)
- bitcoin/bitcoin#6284
- bitcoin/bitcoin#6489
- bitcoin/bitcoin#6235
- bitcoin/bitcoin#6905
- bitcoin/bitcoin#6780
  - Excluding second commit (QT) and third commit (requires bitcoin/bitcoin#6993)
- bitcoin/bitcoin#6961
  - Excluding QT parts, and a small `src/policy/policy.cpp` change which depends on a bunch of other PRs, which we'll have to remember to come back to.
- bitcoin/bitcoin#7044
- bitcoin/bitcoin#8856
- bitcoin/bitcoin#9002

Part of #2074 and #2132.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.