Skip BIP 30 verification where not necessary #6931

Merged
merged 2 commits into from Nov 12, 2015

Conversation

Projects
None yet
10 participants
@morcos
Member

morcos commented Nov 3, 2015

Since BIP 34 became active, it has been impossible to generate duplicate coinbases, and therefore not possible to generate any other duplicate transaction ID's.

This pull will allow us to skip BIP 30 verification if we know we are on a chain after the point BIP 34 was activated and there are no latent duplicate transaction chains.
On mainnet there were 2 cases of duplicate coinbases, and in both cases the duplicate overwrote the original before there was an opportunity to spend it.
I have not actually verified that there are no duplicate coinbases on testnet3, and we need to verify that or turn off the skipping on testnet before merging this pull. (EDIT: verified)

In conjunction with #6932 this will help ConnectBlock be much more efficient with caching access to the database.

@instagibbs

This comment has been minimized.

Show comment
Hide comment
@instagibbs

instagibbs Nov 3, 2015

Member

Do you have a benchmark with both PRs together?

utACK, aside from the testnet3 coinbase issue

Member

instagibbs commented Nov 3, 2015

Do you have a benchmark with both PRs together?

utACK, aside from the testnet3 coinbase issue

@gavinandresen

This comment has been minimized.

Show comment
Hide comment
@gavinandresen

gavinandresen Nov 3, 2015

Contributor

Concept ACK, but the logic seems more complicated than necessary. Couldn't you just remove the BIP30 checking code entirely now that BIP34 has activated? Why do the work in the old chain, when we know the old chain doesn't violate the rules (and have headers-first to prevent somebody from feeding us a low-work bogus chain)?

Contributor

gavinandresen commented Nov 3, 2015

Concept ACK, but the logic seems more complicated than necessary. Couldn't you just remove the BIP30 checking code entirely now that BIP34 has activated? Why do the work in the old chain, when we know the old chain doesn't violate the rules (and have headers-first to prevent somebody from feeding us a low-work bogus chain)?

@sdaftuar

This comment has been minimized.

Show comment
Hide comment
@sdaftuar

sdaftuar Nov 3, 2015

Member

@gavinandresen Headers-first doesn't actually prevent someone from doing the attack -- for example the single peer a node chooses to download headers from could feed a bogus chain before a node ever learns about the real chain.

Concept ACK

Member

sdaftuar commented Nov 3, 2015

@gavinandresen Headers-first doesn't actually prevent someone from doing the attack -- for example the single peer a node chooses to download headers from could feed a bogus chain before a node ever learns about the real chain.

Concept ACK

@morcos

This comment has been minimized.

Show comment
Hide comment
@morcos

morcos Nov 3, 2015

Member

@instagibbs I'm working on putting together some benchmarks. There are a set of pulls that I've been working on to speed up ConnectBlock and CreateNewBlock. I'll post an issue identifying them all and with some benchmarks. But roughly speaking we're talking about these times on average over some recent data for CreateNewBlock (with 1GB dbcache and 1M sigcachesize)

Code version time in ms
Master 550
libsecp256k merged 340
libsecp and these 2 pulls* 160

* I actually didn't do the coinbase lookup required in 6932 for this test, so that can be the difference between 0 and 1 db lookup.

@gavinandresen I'd be interested if you have any thoughts on how best to benchmark this as well?

Member

morcos commented Nov 3, 2015

@instagibbs I'm working on putting together some benchmarks. There are a set of pulls that I've been working on to speed up ConnectBlock and CreateNewBlock. I'll post an issue identifying them all and with some benchmarks. But roughly speaking we're talking about these times on average over some recent data for CreateNewBlock (with 1GB dbcache and 1M sigcachesize)

Code version time in ms
Master 550
libsecp256k merged 340
libsecp and these 2 pulls* 160

* I actually didn't do the coinbase lookup required in 6932 for this test, so that can be the difference between 0 and 1 db lookup.

@gavinandresen I'd be interested if you have any thoughts on how best to benchmark this as well?

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Nov 3, 2015

Member

utACK (will test)

Member

gmaxwell commented Nov 3, 2015

utACK (will test)

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Nov 4, 2015

Member
Member

sipa commented Nov 4, 2015

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Nov 6, 2015

Contributor

Some random stats (generated using #6965):

2015-11-06 23:22:03 UpdateTip: new best=00000000000000000e168ce1bbcdab4551cc35e9d5128b75e5d603143cdc970a height=357552 log2_work=82.822239 tx=69465120 date=2015-05-22 12:42:52 progress=0.799217 cache=1120.9MiB(601684tx)
2015-11-06 23:22:03 - Connect postprocess: 3.40ms [2.65s]
2015-11-06 23:22:03 - Connect block: 328.16ms [230.41s]
2015-11-06 23:22:03 - Load block from disk: 18.28ms [21.80s]
2015-11-06 23:22:03 - Sanity checks: 2.83ms [1.73s]
2015-11-06 23:22:03 - Fork checks: 68.39ms [38.97s]
2015-11-06 23:22:03 - Connect 1501 transactions: 178.28ms (0.119ms/tx, 0.043ms/txin) [92.67s]
2015-11-06 23:22:03 - Verify 4109 txins: 245.83ms (0.060ms/txin) [150.33s]
2015-11-06 23:22:03 - Index writing: 0.04ms [0.06s]
2015-11-06 23:22:03 - Callbacks: 0.03ms [0.03s]
2015-11-06 23:22:03 - Connect total: 318.09ms [191.65s]
2015-11-06 23:22:03 - Flush: 26.23ms [14.01s]
2015-11-06 23:22:03 - Writing chainstate: 2.05ms [0.67s]

2015-11-06 23:32:37 UpdateTip: new best=00000000000000000e168ce1bbcdab4551cc35e9d5128b75e5d603143cdc970a height=357552 log2_work=82.822239 tx=69465120 date=2015-05-22 12:42:52 progress=0.799210 cache=1120.9MiB(601684tx)
2015-11-06 23:32:37 - Connect postprocess: 5.06ms [2.85s]
2015-11-06 23:32:37 - Connect block: 198.12ms [160.29s]
2015-11-06 23:32:37 - Load block from disk: 14.21ms [7.67s]
2015-11-06 23:32:37 - Sanity checks: 2.79ms [1.81s]
2015-11-06 23:32:37 - Fork checks: 0.04ms [0.14s]
2015-11-06 23:32:37 - Connect 1501 transactions: 173.07ms (0.115ms/tx, 0.042ms/txin) [96.87s]
2015-11-06 23:32:37 - Verify 4109 txins: 196.50ms (0.048ms/txin) [131.39s]
2015-11-06 23:32:37 - Index writing: 0.08ms [0.07s]
2015-11-06 23:32:37 - Callbacks: 0.07ms [0.03s]
2015-11-06 23:32:37 - Connect total: 200.48ms [134.03s]
2015-11-06 23:32:37 - Flush: 27.12ms [15.16s]
2015-11-06 23:32:37 - Writing chainstate: 2.27ms [0.83s]

Contributor

TheBlueMatt commented Nov 6, 2015

Some random stats (generated using #6965):

2015-11-06 23:22:03 UpdateTip: new best=00000000000000000e168ce1bbcdab4551cc35e9d5128b75e5d603143cdc970a height=357552 log2_work=82.822239 tx=69465120 date=2015-05-22 12:42:52 progress=0.799217 cache=1120.9MiB(601684tx)
2015-11-06 23:22:03 - Connect postprocess: 3.40ms [2.65s]
2015-11-06 23:22:03 - Connect block: 328.16ms [230.41s]
2015-11-06 23:22:03 - Load block from disk: 18.28ms [21.80s]
2015-11-06 23:22:03 - Sanity checks: 2.83ms [1.73s]
2015-11-06 23:22:03 - Fork checks: 68.39ms [38.97s]
2015-11-06 23:22:03 - Connect 1501 transactions: 178.28ms (0.119ms/tx, 0.043ms/txin) [92.67s]
2015-11-06 23:22:03 - Verify 4109 txins: 245.83ms (0.060ms/txin) [150.33s]
2015-11-06 23:22:03 - Index writing: 0.04ms [0.06s]
2015-11-06 23:22:03 - Callbacks: 0.03ms [0.03s]
2015-11-06 23:22:03 - Connect total: 318.09ms [191.65s]
2015-11-06 23:22:03 - Flush: 26.23ms [14.01s]
2015-11-06 23:22:03 - Writing chainstate: 2.05ms [0.67s]

2015-11-06 23:32:37 UpdateTip: new best=00000000000000000e168ce1bbcdab4551cc35e9d5128b75e5d603143cdc970a height=357552 log2_work=82.822239 tx=69465120 date=2015-05-22 12:42:52 progress=0.799210 cache=1120.9MiB(601684tx)
2015-11-06 23:32:37 - Connect postprocess: 5.06ms [2.85s]
2015-11-06 23:32:37 - Connect block: 198.12ms [160.29s]
2015-11-06 23:32:37 - Load block from disk: 14.21ms [7.67s]
2015-11-06 23:32:37 - Sanity checks: 2.79ms [1.81s]
2015-11-06 23:32:37 - Fork checks: 0.04ms [0.14s]
2015-11-06 23:32:37 - Connect 1501 transactions: 173.07ms (0.115ms/tx, 0.042ms/txin) [96.87s]
2015-11-06 23:32:37 - Verify 4109 txins: 196.50ms (0.048ms/txin) [131.39s]
2015-11-06 23:32:37 - Index writing: 0.08ms [0.07s]
2015-11-06 23:32:37 - Callbacks: 0.07ms [0.03s]
2015-11-06 23:32:37 - Connect total: 200.48ms [134.03s]
2015-11-06 23:32:37 - Flush: 27.12ms [15.16s]
2015-11-06 23:32:37 - Writing chainstate: 2.27ms [0.83s]

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Nov 6, 2015

Contributor

In other words, of some random set of blocks (~356331-357552), this pull took the ConnectBlock time from 230s to 160s!

Contributor

TheBlueMatt commented Nov 6, 2015

In other words, of some random set of blocks (~356331-357552), this pull took the ConnectBlock time from 230s to 160s!

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Nov 7, 2015

Member

Concept & code review ACK.

Member

sipa commented Nov 7, 2015

Concept & code review ACK.

@petertodd

This comment has been minimized.

Show comment
Hide comment
@petertodd

petertodd Nov 10, 2015

Contributor

utACK modulo duplicate testnet coinbases issue.

Contributor

petertodd commented Nov 10, 2015

utACK modulo duplicate testnet coinbases issue.

@petertodd

This comment has been minimized.

Show comment
Hide comment
@petertodd

petertodd Nov 10, 2015

Contributor

Just confirmed that all coinbases from block #0 to block #21112 on testnet3 are unique.

Also successfully did a sync from scratch on testnet3.

ACK

Contributor

petertodd commented Nov 10, 2015

Just confirmed that all coinbases from block #0 to block #21112 on testnet3 are unique.

Also successfully did a sync from scratch on testnet3.

ACK

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Nov 11, 2015

Member

Untested ACK

Member

sipa commented Nov 11, 2015

Untested ACK

@morcos

This comment has been minimized.

Show comment
Hide comment
@morcos

morcos Nov 12, 2015

Member

Also confirmed unique coinbases on tesnet3.

Member

morcos commented Nov 12, 2015

Also confirmed unique coinbases on tesnet3.

@sipa sipa merged commit 33c90cf into bitcoin:master Nov 12, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

sipa added a commit that referenced this pull request Nov 12, 2015

Merge pull request #6931
33c90cf Make skipping BIP30 check chain agnostic (Alex Morcos)
06d81ad Skip BIP30 check after BIP34 activation (Alex Morcos)

@Infernoman Infernoman referenced this pull request in Crowndev/crown-core Sep 25, 2016

Merged

Skip BIP 30 verification where not necessary #6931 #21

FireWalkerX added a commit to Crowndev/crown-core that referenced this pull request Oct 11, 2016

Merge pull request #21 from Infernoman/patch-12
Skip BIP 30 verification where not necessary #6931
@evoskuil

This comment has been minimized.

Show comment
Hide comment
@evoskuil

evoskuil Dec 11, 2016

Once BIP34 activated it was not possible to create new duplicate coinbases and thus other than starting with the 2 existing duplicate coinbase pairs, not possible to create overwriting txs.

This is an incorrect assumption. BIP30 addresses hash collision. It is possible for transaction hashes to collide through nothing more than happenstance.

If we're on the known chain at height greater than 227931 where BIP34 activated, we can save the db accesses needed for the BIP30 check.

BIP34 does not address the real possibility of transaction hash collision, it only addresses duplicate transactions. Eliminating the BIP30 check is a hard fork that will result in the arbitrary destruction of unspent outputs.

Incidentally, the implementation assumes incorrectly that BIP34 must have activated in the case where a checkpoint at which BIP34 previously activated is found to be present. It is possible, however unlikely, that the checkpoint represents a different history, again owing to the possibility of hash collision. Checkpoints are optimizations and should never be used for security.

evoskuil commented on 06d81ad Dec 11, 2016

Once BIP34 activated it was not possible to create new duplicate coinbases and thus other than starting with the 2 existing duplicate coinbase pairs, not possible to create overwriting txs.

This is an incorrect assumption. BIP30 addresses hash collision. It is possible for transaction hashes to collide through nothing more than happenstance.

If we're on the known chain at height greater than 227931 where BIP34 activated, we can save the db accesses needed for the BIP30 check.

BIP34 does not address the real possibility of transaction hash collision, it only addresses duplicate transactions. Eliminating the BIP30 check is a hard fork that will result in the arbitrary destruction of unspent outputs.

Incidentally, the implementation assumes incorrectly that BIP34 must have activated in the case where a checkpoint at which BIP34 previously activated is found to be present. It is possible, however unlikely, that the checkpoint represents a different history, again owing to the possibility of hash collision. Checkpoints are optimizations and should never be used for security.

cerebrus29301 pushed a commit to cerebrus29301/crowncoin that referenced this pull request Feb 4, 2017

Skip BIP 30 verification where not necessary #6931
bitcoin#6931
BIP-0034 should've been active by the time crown started.
PLEASE TEST.
Former-commit-id: c765f36

cerebrus29301 pushed a commit to cerebrus29301/crowncoin that referenced this pull request Feb 4, 2017

Merge pull request #21 from Infernoman/patch-12
Skip BIP 30 verification where not necessary #6931

Former-commit-id: dbea57a

@dagurval dagurval referenced this pull request in bitcoinxt/bitcoinxt Mar 14, 2017

Merged

Chainparams backports #189

laanwj added a commit that referenced this pull request Mar 7, 2018

Merge #12204: Fix overly eager BIP30 bypass
5b8b387 Fix overly eager BIP30 bypass (Alex Morcos)

Pull request description:

  In #6931 we introduced a possible consensus breaking change by misunderstanding how completely BIP 34 obviated the need for BIP 30.  Unfixed, this could break consensus after block height about 1.9M.  Explained in code comment.

  h/t @sdaftuar

Tree-SHA512: 8f798c3f203432fd4ae1c1c08bd6967b4a5ec2064ed5f6a7dcf3bff34ea830952838dd4ff70d70b5080cf4644f601e5526b60456c08f43789e4aae05621d9d6b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment