Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Rewrite DoS interface between validation and net_processing #15141
This is a rebase of #11639 with some fixes for the last few comments which were not yet addressed.
The original PR text, with some strikethroughs of text that is no longer correct:
Note: The change to ban all peers for consensus violations is actually NOT the change I'd like to make -- I'd prefer to only ban outbound peers in those situations. The current behavior is a bit of a mess, however, and so in the interests of advancing this PR I tried to keep the changes to a minimum. I plan to revisit the behavior in a followup PR.
EDIT: One reviewer suggested I add some additional context for this PR:
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.
Reviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.
ryanofsky left a comment
Started reviewing this, but IMO, the way this PR is structured makes it difficult to verify that it doesn't unintentionally change behavior.
I think a nicer way to write this would be to have one commit adding empty
If this is a bad idea, or too much work, I'd be ok with trying to review this PR as it is, but I wanted to suggest something to be able to have more confidence in it, and to maybe make it easier to find other reviewers.
FWIW, I've had a go at redoing the patchset to try to make the (potential) functionality changes more clear: https://github.com/ajtowns/bitcoin/commits/201901-dosreasons
This has (I think) all the behaviour changes first:
before introducing the new reason field, along with checks that the implied DoS value for each reason matches the actual DoS values presented/used:
which then allows dropping the instance variables:
Then the code is changed to use reasons directly:
And the now obsolete DoS/etc stuff is dropped:
That leaves a couple more things:
but finally ends up with the same code as this PR (minus the latest commit anyway).
Anyway I think this approach might be easier to review? It could also allow splitting the PR into two -- one making the changes to DoS behaviour but not changing the way DoS works; followed by a second PR that actually adds the Reasons and refactors but doesn't change behaviour.
(Proof of concept only: bunches of these commits should probably be combined, commit messages need improvement, and I think I lost a bunch of authorship info)
I've added an extra commit prior to the DoS->Invalid refactor, namely "5b15205883 Allow use of state.Invalid() for all reasons" that avoids assertions that Invalid() is only used for DoS-level-0 problems failing.
That just leaves one test failure in the intermediate commits; feature_block.py fails after the changing the DoS levels but before adding the "reason" code. I think this is due to lowering
Thanks all for the review so far!
I'd started taking a stab at rewriting this; I'll continue with my approach to see how it ends up but @ajtowns thank you for your help -- @ryanofsky if you have any thoughts on @ajtowns's rework please let me know, happy to adapt his breakdown and include here if that approach looks good.
Took a quick look, and I think ajtowns's refactor is great. It's a slightly different approach than I suggested in that the 32747d0 commit which starts using reason codes is done all at once instead of incrementally as reasons are added, so it requires a little bit of grepping to verify, but this is easy to do and I think it's a huge improvement.
I think it would be best to use ajtown's branch here, unless you've done a lot of work on your own already or see problems I'm missing.
I have redone this along the lines of @ajtowns branch, and cleaned up each commit (I think!) so that each one should be logically correct, pass tests, etc.
I've saved the original version of this PR here: https://github.com/sdaftuar/bitcoin/commits/15141.original
The diff between the two is pretty small (just some formatting changes that were getting tedious to resolve, and I removed a couple lines that some reviewers had commented on as being unnecessary):
Also if this version is not actually easier to review I'm happy to go back to the original or try another approach.
ryanofsky left a comment •
Started review (will update list below with progress).
I addressed @ryanofsky's comments so far (which rewrote the git history, since one of the commit messages changed, so I also squashed in a comment change as well). Previous version of this PR is now here: https://github.com/sdaftuar/bitcoin/commits/15141.1.
One overall comment: it seems there is a subset of
Compared with previous bans, the following changes are made: * Txn with empty vin/vout or null prevouts move from 10 DoS points to 100. * Loose transactions with a dependency loop now result in a ban instead of 10 DoS points. * Many pre-segwit soft-fork errors now result in a ban. Note: Transactions that violate soft-fork script flags since P2SH do not generally result in a ban. Also, banning behavior for invalid blocks is dependent on whether the node is validating with multiple script check threads, due to a long- standing bug. That inconsistency is still present after this commit. * Proof of work failure moves from 50 DoS points to a ban. * Blocks with timestamps under MTP now result in a ban, blocks too far in the future continue to *not* result in a ban. * Inclusion of non-final transactions in a block now results in a ban instead of 10 DoS points. Co-authored-by: Anthony Towns <firstname.lastname@example.org>
This eliminates a discrepancy between block validation with multiple script check threads, versus a single script check thread.
This is a first step towards cleaning up our DoS interface - make validation return *why* something is invalid, and let net_processing figure out what that implies in terms of banning/disconnection/etc. Behavior change: peers will now be banned for providing blocks with premature coinbase spends. Co-authored-by: Anthony Towns <email@example.com> Suhas Daftuar <firstname.lastname@example.org>
We only disconnect outbound peers (excluding HB compact block peers and manual connections) when receiving a CACHED_INVALID header.
-BEGIN VERIFY SCRIPT- sed -i 's/\.DoS(\(.*\), REJECT_\(.*\), \(true\|false\)/.DoS(\1, REJECT_\2/' src/validation.cpp src/consensus/tx_verify.cpp src/consensus/tx_check.cpp sed -i 's/state.GetRejectCode(), state.GetRejectReason(), [^,]\+, state.GetDebugMessage())/state.GetRejectCode(), state.GetRejectReason(), state.GetDebugMessage())/' src/validation.cpp sed -i 's/\.DoS([^,]*, /.Invalid\(/' src/validation.cpp src/consensus/tx_verify.cpp src/consensus/tx_check.cpp -END VERIFY SCRIPT- Co-authored-by: Suhas Daftuar <email@example.com>
…sing 0ff1c2a Separate reason for premature spends (coinbase/locktime) (Suhas Daftuar) 54470e7 Assert validation reasons are contextually correct (Suhas Daftuar) 2120c31 [refactor] Update some comments in validation.cpp as we arent doing DoS there (Matt Corallo) 12dbdd7 [refactor] Drop unused state.DoS(), state.GetDoS(), state.CorruptionPossible() (Matt Corallo) aa502b8 scripted-diff: Remove DoS calls to CValidationState (Matt Corallo) 7721ad6 [refactor] Prep for scripted-diff by removing some \ns which annoy sed. (Matt Corallo) 5e78c57 Allow use of state.Invalid() for all reasons (Matt Corallo) 6b34bc6 Fix handling of invalid headers (Suhas Daftuar) ef54b48 [refactor] Use Reasons directly instead of DoS codes (Matt Corallo) 9ab2a04 CorruptionPossible -> BLOCK_MUTATED (Matt Corallo) 6e55b29 CorruptionPossible -> TX_WITNESS_MUTATED (Matt Corallo) 7df16e7 LookupBlockIndex -> CACHED_INVALID (Matt Corallo) c8b0d22 [refactor] Drop redundant nDoS, corruptionPossible, SetCorruptionPossible (Matt Corallo) 34477cc [refactor] Add useful-for-dos "reason" field to CValidationState (Matt Corallo) 6a7f877 Ban all peers for all block script failures (Suhas Daftuar) 7b99910 Clean up banning levels (Matt Corallo) b8b4c80 [refactor] drop IsInvalid(nDoSOut) (Matt Corallo) 8818729 [refactor] Refactor misbehavior ban decisions to MaybePunishNode() (Matt Corallo) 00e11e6 [refactor] rename stateDummy -> orphan_state (Matt Corallo) f34fa71 Drop obsolete sigops comment (Matt Corallo) Pull request description: This is a rebase of #11639 with some fixes for the last few comments which were not yet addressed. The original PR text, with some strikethroughs of text that is no longer correct: > This cleans up an old main-carryover - it made sense that main could decide what DoS scores to assign things because the DoS scores were handled in a different part of main, but now validation is telling net_processing what DoS scores to assign to different things, which is utter nonsense. Instead, we replace CValidationState's nDoS and CorruptionPossible with a general ValidationInvalidReason, which net_processing can handle as it sees fit. I keep the behavior changes here to a minimum, but in the future we can utilize these changes for other smarter behavior, such as disconnecting/preferring to rotate outbound peers based on them providing things which are invalid due to SOFT_FORK because we shouldn't ban for such cases. > > This is somewhat complementary with, though obviously conflicts heavily with #11523, which added enums in place of DoS scores, as well as a few other cleanups (which are still relevant). > > Compared with previous bans, the following changes are made: > > Txn with empty vin/vout or null prevouts move from 10 DoS > points to 100. > Loose transactions with a dependency loop now result in a ban > instead of 10 DoS points. > ~~BIP68-violation no longer results in a ban as it is SOFT_FORK.~~ > ~~Non-SegWit SigOp violation no longer results in a ban as it > considers P2SH sigops and is thus SOFT_FORK.~~ > ~~Any script violation in a block no longer results in a ban as > it may be the result of a SOFT_FORK. This should likely be > fixed in the future by differentiating between them.~~ > Proof of work failure moves from 50 DoS points to a ban. > Blocks with timestamps under MTP now result in a ban, blocks > too far in the future continue to not result in a ban. > Inclusion of non-final transactions in a block now results in a > ban instead of 10 DoS points. Note: The change to ban all peers for consensus violations is actually NOT the change I'd like to make -- I'd prefer to only ban outbound peers in those situations. The current behavior is a bit of a mess, however, and so in the interests of advancing this PR I tried to keep the changes to a minimum. I plan to revisit the behavior in a followup PR. EDIT: One reviewer suggested I add some additional context for this PR: > The goal of this work was to make net_processing aware of the actual reasons for validation failures, rather than just deal with opaque numbers instructing it to do something. > > In the future, I'd like to make it so that we use more context to decide how to punish a peer. One example is to differentiate inbound and outbound peer misbehaviors. Another potential example is if we'd treat RECENT_CONSENSUS_CHANGE failures differently (ie after the next consensus change is implemented), and perhaps again we'd want to treat some peers differently than others. ACKs for commit 0ff1c2: jnewbery: utACK 0ff1c2a ryanofsky: utACK 0ff1c2a. Only change is dropping the first commit (f3883a3), and dropping the temporary `assert(level == GetDoS())` that was in 35ee77f (now c8b0d22) Tree-SHA512: e915a411100876398af5463d0a885920e44d473467bb6af991ef2e8f2681db6c1209bb60f848bd154be72d460f039b5653df20a6840352c5f7ea5486d9f777a3