New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove duplicatable duplicate-input check from CheckTransaction #9049

Merged
merged 3 commits into from Nov 10, 2016

Conversation

Projects
None yet
@TheBlueMatt
Contributor

TheBlueMatt commented Oct 31, 2016

Benchmark results indicate this saves about 0.5-0.7ms during CheckBlock.

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Nov 1, 2016

Member

I believe that with this change if a whitelisted peer gives us a transaction with internally duplicated inputs, we will relay it (CheckTransaction will pass, conflict with the mempool will not set DoS.) and then our peers running without this change will ban us (Their CheckTransaction will fail). Less critically, this change results in txn with internal doublespends getting a misleading log entry of mempool-conflict.

Edit: I suggest creating a CheckTransactionHarder that ATMP uses which will check for transaction internal duplication and still nDoS. This way block processing can still avoid the redundant test.

Member

gmaxwell commented Nov 1, 2016

I believe that with this change if a whitelisted peer gives us a transaction with internally duplicated inputs, we will relay it (CheckTransaction will pass, conflict with the mempool will not set DoS.) and then our peers running without this change will ban us (Their CheckTransaction will fail). Less critically, this change results in txn with internal doublespends getting a misleading log entry of mempool-conflict.

Edit: I suggest creating a CheckTransactionHarder that ATMP uses which will check for transaction internal duplication and still nDoS. This way block processing can still avoid the redundant test.

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Nov 1, 2016

Contributor

Simplified diff and fixed.

Contributor

TheBlueMatt commented Nov 1, 2016

Simplified diff and fixed.

if (vInOutPoints.count(txin.prevout))
return state.DoS(100, false, REJECT_INVALID, "bad-txns-inputs-duplicate");
vInOutPoints.insert(txin.prevout);
// Check for duplicate inputs - note that this check is slow so we skip it in CheckBlock

This comment has been minimized.

@gmaxwell

gmaxwell Nov 2, 2016

Member

point out that it's also redundant there please!

@gmaxwell

gmaxwell Nov 2, 2016

Member

point out that it's also redundant there please!

This comment has been minimized.

@dexX7

dexX7 Sep 19, 2018

Contributor

How is it redundant? Can you point to the case where it's checked a second time?

@dexX7

dexX7 Sep 19, 2018

Contributor

How is it redundant? Can you point to the case where it's checked a second time?

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Nov 2, 2016

Member

utACK +/- comment nit.

Member

gmaxwell commented Nov 2, 2016

utACK +/- comment nit.

Show outdated Hide outdated src/bench/checkblock.cpp Outdated

@laanwj laanwj added the Tests label Nov 2, 2016

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Nov 2, 2016

Contributor

Eek, license change was a mistake, sorry.

On November 2, 2016 5:29:28 AM EDT, "Wladimir J. van der Laan" notifications@github.com wrote:

laanwj requested changes on this pull request.

@@ -0,0 +1,53 @@
+// Copyright (c) 2016 Matt Corallo
+// Unlike the rest of Bitcoin Core, this file is
+// distributed under the Affero General Public License (AGPL v3)

Code under this license cannot be accepted, sorry

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#9049 (review)

Contributor

TheBlueMatt commented Nov 2, 2016

Eek, license change was a mistake, sorry.

On November 2, 2016 5:29:28 AM EDT, "Wladimir J. van der Laan" notifications@github.com wrote:

laanwj requested changes on this pull request.

@@ -0,0 +1,53 @@
+// Copyright (c) 2016 Matt Corallo
+// Unlike the rest of Bitcoin Core, this file is
+// distributed under the Affero General Public License (AGPL v3)

Code under this license cannot be accepted, sorry

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#9049 (review)

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Nov 2, 2016

Contributor

Fixed the broken license header.

Contributor

TheBlueMatt commented Nov 2, 2016

Fixed the broken license header.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Nov 3, 2016

Member

A very interesting duplication of the word "duplicate" in the PR title.

Member

sipa commented Nov 3, 2016

A very interesting duplication of the word "duplicate" in the PR title.

@MarcoFalke MarcoFalke changed the title from Remove duplicatble duplicate-input check from CheckTransaction to Remove duplicatable duplicate-input check from CheckTransaction Nov 3, 2016

@MarcoFalke MarcoFalke added Resource usage Tests and removed Tests labels Nov 7, 2016

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Nov 7, 2016

Member

@laanwj I believe you need to dismiss the requested changes (as they're addressed).

Member

sipa commented Nov 7, 2016

@laanwj I believe you need to dismiss the requested changes (as they're addressed).

@laanwj

laanwj approved these changes Nov 8, 2016

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Nov 8, 2016

Member

Approved the requested change.
BTW: block413567.hex.h is a 6MB file - I'd prefer generating this on the fly as we do with the other .X.h files.
Edit: working on build system change for this...

Member

laanwj commented Nov 8, 2016

Approved the requested change.
BTW: block413567.hex.h is a 6MB file - I'd prefer generating this on the fly as we do with the other .X.h files.
Edit: working on build system change for this...

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Nov 8, 2016

Member

Can you please squash the top commit of https://github.com/laanwj/bitcoin/tree/2016_11_generate_bench_file into the benchmark commit?

This has the necessary build system change to generate the block.hex file on the fly from the raw data, like with the tests.

@theuni can you have take a look too perhaps?

Member

laanwj commented Nov 8, 2016

Can you please squash the top commit of https://github.com/laanwj/bitcoin/tree/2016_11_generate_bench_file into the benchmark commit?

This has the necessary build system change to generate the block.hex file on the fly from the raw data, like with the tests.

@theuni can you have take a look too perhaps?

@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Nov 8, 2016

Member

@laanwj looks good to me. Only slight criticism is that we may want to generate .h/.cpp pairs for blocks to ensure that they end up in separate compilation units, but we can always address that later if it becomes an issue.

Member

theuni commented Nov 8, 2016

@laanwj looks good to me. Only slight criticism is that we may want to generate .h/.cpp pairs for blocks to ensure that they end up in separate compilation units, but we can always address that later if it becomes an issue.

Show outdated Hide outdated src/main.cpp Outdated
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Nov 9, 2016

Member

@laanwj looks good to me. Only slight criticism is that we may want to generate .h/.cpp pairs for blocks to ensure that they end up in separate compilation units, but we can always address that later if it becomes an issue.

Fully agree. On the long run, I'd like to forego the conversion to hex and subsequent compilation entirely on build chains that supports this, by embedding the binary directly as mentioned in a comment here #6658 (comment). The current way wastes a lot of time and memory while compiling.

Member

laanwj commented Nov 9, 2016

@laanwj looks good to me. Only slight criticism is that we may want to generate .h/.cpp pairs for blocks to ensure that they end up in separate compilation units, but we can always address that later if it becomes an issue.

Fully agree. On the long run, I'd like to forego the conversion to hex and subsequent compilation entirely on build chains that supports this, by embedding the binary directly as mentioned in a comment here #6658 (comment). The current way wastes a lot of time and memory while compiling.

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Nov 9, 2016

Contributor

Picked up (and squashed) @laanwj's changes and went ahead and remove the duplicate-lookup for ATMP uses.

Contributor

TheBlueMatt commented Nov 9, 2016

Picked up (and squashed) @laanwj's changes and went ahead and remove the duplicate-lookup for ATMP uses.

@laanwj laanwj merged commit e2b3fb3 into bitcoin:master Nov 10, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Nov 10, 2016

Merge #9049: Remove duplicatable duplicate-input check from CheckTran…
…saction

e2b3fb3 Optimize vInOutPoints insertion a bit (Matt Corallo)
eecffe5 Remove redundant duplicate-input check from CheckTransaction (Matt Corallo)
b2e178a Add deserialize + CheckBlock benchmarks, and a full block hex (Matt Corallo)

paveljanik added a commit to paveljanik/bitcoin that referenced this pull request Nov 10, 2016

Use better name for local variable to prevent -Wshadow compiler warning.
bitcoin#9049 brought new -Wshadow warning. Local variable shadows the function
argument. I decided to keep the function argument name to keep konsistent with
the rest of the file and use better/descriptive name for the local variable.

codablock added a commit to codablock/dash that referenced this pull request Jan 15, 2018

Merge bitcoin#9049: Remove duplicatable duplicate-input check from Ch…
…eckTransaction

e2b3fb3 Optimize vInOutPoints insertion a bit (Matt Corallo)
eecffe5 Remove redundant duplicate-input check from CheckTransaction (Matt Corallo)
b2e178a Add deserialize + CheckBlock benchmarks, and a full block hex (Matt Corallo)

TheBlueMatt pushed a commit to TheBlueMatt/bitcoin that referenced this pull request Sep 17, 2018

CryptoDJ added a commit to CryptoDJ/BitcoinDiamond that referenced this pull request Sep 22, 2018

CryptoDJ added a commit to CryptoDJ/syscoin that referenced this pull request Sep 22, 2018

CryptoDJ added a commit to CryptoDJ/namecoin-core that referenced this pull request Sep 22, 2018

CryptoDJ added a commit to CryptoDJ/GameCredits that referenced this pull request Sep 22, 2018

CryptoDJ added a commit to CryptoDJ/Feathercoin that referenced this pull request Sep 22, 2018

CryptoDJ added a commit to CryptoDJ/syscoin that referenced this pull request Sep 22, 2018

CryptoDJ added a commit to CryptoDJ/huntercore that referenced this pull request Sep 22, 2018

@ripper234

This comment has been minimized.

Show comment
Hide comment
@ripper234

ripper234 Sep 22, 2018

So, if I understand correctly, this caused CVE-2018-17144.

Can we do a root cause analysis / 5 whys explaining how this was introduced?

ripper234 commented Sep 22, 2018

So, if I understand correctly, this caused CVE-2018-17144.

Can we do a root cause analysis / 5 whys explaining how this was introduced?

@thijstriemstra

This comment has been minimized.

Show comment
Hide comment
@thijstriemstra

thijstriemstra Sep 22, 2018

It's scary to see how this code was merged with so much confidence. The fact there wasn't a unit test covering this piece of code makes me feel little better though, a bad unit test would be even worse. I hope btc core will never accept contributions without tests again.

thijstriemstra commented Sep 22, 2018

It's scary to see how this code was merged with so much confidence. The fact there wasn't a unit test covering this piece of code makes me feel little better though, a bad unit test would be even worse. I hope btc core will never accept contributions without tests again.

@dfoderick

This comment has been minimized.

Show comment
Hide comment
@dfoderick

dfoderick Sep 22, 2018

Can we do a root cause analysis / 5 whys explaining how this was introduced?

Short version.
This pull request added a boolean parameter to CheckTransaction method
bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fCheckDuplicateInputs)

Then in CheckBlock method the flag was passed in as false bypassing the check for duplicate inputs. Assumption was that transaction would already be validated by other code so there was no need to validate a second time.
if (!CheckTransaction(tx, state, false))

Why there was no unit test?
Why there are so many assumptions in code?
Why did programmer not put the flag on transaction class instead?

dfoderick commented Sep 22, 2018

Can we do a root cause analysis / 5 whys explaining how this was introduced?

Short version.
This pull request added a boolean parameter to CheckTransaction method
bool CheckTransaction(const CTransaction& tx, CValidationState &state, bool fCheckDuplicateInputs)

Then in CheckBlock method the flag was passed in as false bypassing the check for duplicate inputs. Assumption was that transaction would already be validated by other code so there was no need to validate a second time.
if (!CheckTransaction(tx, state, false))

Why there was no unit test?
Why there are so many assumptions in code?
Why did programmer not put the flag on transaction class instead?

@ripper234

This comment has been minimized.

Show comment
Hide comment
@ripper234

ripper234 Sep 23, 2018

Why this passed code review? (How many people reviewed it?)

ripper234 commented Sep 23, 2018

Why this passed code review? (How many people reviewed it?)

@adlai

This comment has been minimized.

Show comment
Hide comment
@adlai

adlai Sep 23, 2018

@ripper234 commented 5 hours ago

Why this passed code review? (How many people reviewed it?) [sic]

I'm sure Microsoft is the right corporate person to ask.

adlai commented Sep 23, 2018

@ripper234 commented 5 hours ago

Why this passed code review? (How many people reviewed it?) [sic]

I'm sure Microsoft is the right corporate person to ask.

@ripper234

This comment has been minimized.

Show comment
Hide comment
@ripper234

ripper234 commented Sep 23, 2018

?

@scravy

This comment has been minimized.

Show comment
Hide comment
@scravy

scravy Sep 24, 2018

Contributor

@dfoderick I question that these questions are in the spirit of a 5 whys root-cause analysis. You are supposed to go deeper, not wider as you do.

Anyhow, the PR merged here was a refactoring and performance improvement, and there already was no test that could have been broken by this; which I believe to be the real culprit.

Contributor

scravy commented Sep 24, 2018

@dfoderick I question that these questions are in the spirit of a 5 whys root-cause analysis. You are supposed to go deeper, not wider as you do.

Anyhow, the PR merged here was a refactoring and performance improvement, and there already was no test that could have been broken by this; which I believe to be the real culprit.

@dfoderick

This comment has been minimized.

Show comment
Hide comment
@dfoderick

dfoderick Sep 24, 2018

Anyhow, the PR merged here was a refactoring and performance improvement, and there already was no test that could have been broken by this; which I believe to be the real culprit.

Correct. The test for this bug did not exist at the time and was only recently added.
9b4a36e

There was a bench test to prove the performance gain but no test for correctness. The bug allows a block with an invalid tx to pass block validation.

dfoderick commented Sep 24, 2018

Anyhow, the PR merged here was a refactoring and performance improvement, and there already was no test that could have been broken by this; which I believe to be the real culprit.

Correct. The test for this bug did not exist at the time and was only recently added.
9b4a36e

There was a bench test to prove the performance gain but no test for correctness. The bug allows a block with an invalid tx to pass block validation.

willyko added a commit to syscoin/syscoin that referenced this pull request Sep 24, 2018

@adlai

This comment has been minimized.

Show comment
Hide comment
@adlai

adlai Sep 25, 2018

@ripper234 commented 2 days ago

?

I'm gonna assume your eloquence was directed at me; if you're not the antecedent of that pronoun, or my assumption was asinine, please ignore this message!

@ripper234, if you truly care about "why this passed code review" (See how here the grammar is correct, and there's no need for editorialization? Amazing, this whole language business!), then you should be able to assign a lower bound to "How many people reviewed it" with confidence equal to your mnemonic fidelity. If this message is unclear, please go back to code review, and leave the Github issue brigading to the professional trolls (or perhaps, take your concerns to the mailing list).

adlai commented Sep 25, 2018

@ripper234 commented 2 days ago

?

I'm gonna assume your eloquence was directed at me; if you're not the antecedent of that pronoun, or my assumption was asinine, please ignore this message!

@ripper234, if you truly care about "why this passed code review" (See how here the grammar is correct, and there's no need for editorialization? Amazing, this whole language business!), then you should be able to assign a lower bound to "How many people reviewed it" with confidence equal to your mnemonic fidelity. If this message is unclear, please go back to code review, and leave the Github issue brigading to the professional trolls (or perhaps, take your concerns to the mailing list).

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 25, 2018

Member

I suggest we lock this PR. For all the drive-by commenters two years after the fact: if you're serious about wanting to improve security and code quality in Bitcoin Core, you can start by:

  • testing release candidates
  • reviewing open PRs
  • contributing code
Member

jnewbery commented Sep 25, 2018

I suggest we lock this PR. For all the drive-by commenters two years after the fact: if you're serious about wanting to improve security and code quality in Bitcoin Core, you can start by:

  • testing release candidates
  • reviewing open PRs
  • contributing code

@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 25, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.