Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

p2p: Remove BIP61 reject messages #15437

Merged
merged 1 commit into from Oct 9, 2019
Merged

Conversation

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Feb 18, 2019

Reject messages (BIP 61) appear in the following settings:

  • Parsing of reject messages (in case -debug=net is set, off by default). This has only been used for a single LogPrint call for several releases now. Such logging is completely meaningless to us and should thus be removed.

  • The sending of reject messages (in case -enablebip61 is set, off by default). This can be used to debug a node that is under our control. Instead of hacking this debugging into the p2p protocol, it could be more easily achieved by parsing the debug log. (Use -printtoconsole to have it as stream, or read from the debug.log file like our python function assert_debug_log in the test framework does)

Having to maintain all of this logic and code to accommodate debugging, which can be achieved by other means a lot easier, is a burden. It makes review on net processing changes a lot harder, since the reject message logic has to be carried around without introducing any errors or DOS vectors.

@MarcoFalke MarcoFalke added the P2P label Feb 18, 2019
@MarcoFalke MarcoFalke added this to the 0.19.0 milestone Feb 18, 2019
@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Feb 18, 2019

Not for 0.18.0, obviously

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Feb 18, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #17004 (validation: Remove REJECT code from CValidationState by jnewbery)
  • #16839 (Replace Connman and BanMan globals with Node local by ryanofsky)
  • #16762 (Rust-based Backup over-REST block downloader by TheBlueMatt)
  • #16442 (Serve BIP 157 compact filters by jimpo)
  • #16202 (Refactor network message deserialization by jonasschnelli)
  • #15921 (validation: Tidy up ValidationState interface by jnewbery)
  • #15206 (Immediately disconnect on invalid net message checksum by jonasschnelli)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@MarcoFalke MarcoFalke force-pushed the Mf1809-noBip61 branch 2 times, most recently from fa68bce to fa1fb01 Feb 18, 2019
@practicalswift
Copy link
Contributor

@practicalswift practicalswift commented Feb 19, 2019

Concept ACK

-134 lines: nice!

Copy link
Member

@jnewbery jnewbery left a comment

Mild concept ACK.

However, I think this PR should be preceded by discussion on the dev mailing list. It's possible that organizations are using bitcoind as their border node, and broadcast transactions from internal applications via bitcoind. Perhaps they rely on REJECT messages? @laanwj has also commented that he found the messages indispensable when writing client software: #13134 (comment).

test/functional/feature_dersig.py Outdated Show resolved Hide resolved
@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Feb 20, 2019

Mild concept ACK-- the loc reduction makes a case for it. If someone were relying on this, I'd really encourage them to not do so, unrelated to if this PR were going in or not. :) The ability to use it during testing/debugging, makes for a case against it.

@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Feb 22, 2019

Indeed, if someone relies on this in production, I'd rather have them to switch to safer ways of achieving their goal. So imo another reason to remove the "feature"

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Feb 22, 2019

I'd rather have them to switch to safer ways of achieving their goal. So imo another reason to remove the "feature"

I think there's a difference between encouraging users to change behaviour and pulling a feature out from under them. We really don't have a good picture of how users use our code, so I'd always err on the side of caution when deprecating features.

I'm going to modify my mild concept ACK with a merge-NACK-until-this-has-been-more-widely-discussed. We're probably fine, but it seems to me that any P2P change that was added to Bitcoin Core with a BIP should be removed after public discussion.

Here's how I expect it'll go: someone will post a mail to the bitcoin-core mailing list saying "I propose to remove REJECT messages in Bitcoin Core v0.19", no-one will respond, and then we'll remove them. I'm happy to send that post if you'd prefer not to.

@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Feb 22, 2019

I am happy to write release notes and a mailing list post, I just didn't come around to do this yet.

The previous release 0.18.0 already disabled BIP61 by default, so this shouldn't come off as a massive surprise. Also, I wouldn't be surprised if the entity that uses the feature isn't subscribed to the mailing list. Could Bitcoin Optech send out a mail as well?

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Feb 22, 2019

Thanks Marco. Sounds good. I'm very happy to review the post/release notes if you want.

The previous release 0.18.0 already disabled BIP61 by default, so this shouldn't come off as a massive surprise

Agree. 👍 for staged deprecation of features!

Could Bitcoin Optech send out a mail as well?

We'd certainly include it in our next newsletter if there was a post on the mailing list.

@MarcoFalke MarcoFalke changed the title p2p: Remove remote debugging code p2p: Remove BIP61 reject messages Feb 25, 2019
@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Feb 25, 2019

Added a commit with the draft for the mailing list post as requested by @jnewbery

@MarcoFalke MarcoFalke force-pushed the Mf1809-noBip61 branch 3 times, most recently from 1bbf4ca to 91d1930 Feb 25, 2019
Copy link
Member

@jnewbery jnewbery left a comment

Email draft looks good Marco.

relies on this feature and you can not use any of the recommended alternatives:

* Testing or debugging of implementations of the Bitcoin P2P network protocol
should be done by inspecting the reject messages that are produced by a
Copy link
Member

@jnewbery jnewbery Feb 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this say "inspecting the logs ..."?

Might be useful to have an example log here.

Copy link
Member Author

@MarcoFalke MarcoFalke Feb 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess an example log depends on the message type that was rejected "version", "tx" or "block" and the reason for rejection. I'd rather not add a specific example, since the logs are not guaranteed to be stable (only used for testing and debugging)

Copy link
Member

@jnewbery jnewbery Feb 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it'd still be helpful, even with a caveat that log messages are not guaranteed to be stable. Obviously up to you!

@MarcoFalke MarcoFalke force-pushed the Mf1809-noBip61 branch 2 times, most recently from f22e760 to 3458603 Feb 25, 2019
@MarcoFalke MarcoFalke removed this from the 0.19.0 milestone Mar 14, 2019
@MarcoFalke MarcoFalke added this to the 0.20.0 milestone Mar 14, 2019
@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Mar 14, 2019

Moved to 0.20.0 milestone for now

Copy link
Member

@jnewbery jnewbery left a comment

A few minor comments inline. Otherwise ACK fab8741

(for merging after the 0.19 branch)

src/net_processing.cpp Show resolved Hide resolved
src/net_processing.cpp Show resolved Hide resolved
test/functional/feature_dersig.py Outdated Show resolved Hide resolved
@jnewbery
Copy link
Member

@jnewbery jnewbery commented Sep 30, 2019

Thanks for the clarifications. ACK once the test comment is corrected.

doc/bips.md Show resolved Hide resolved
@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Oct 2, 2019

Addressed doc nits by @jnewbery and @laanwj

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Oct 3, 2019

utACK fa25f43

@laanwj
Copy link
Member

@laanwj laanwj commented Oct 3, 2019

I'm still not 100% convinced that I like getting rid of BIP61 conceptually, but apparently everyone wants it, code review ACK fa25f43.

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Code review ACK fa25f43

I don't know much about the BIP process, but should https://github.com/bitcoin/bips/blob/master/bip-0061.mediawiki have some kind of "Status: Obsolete" tag or a separate warning that it shouldn't be relied on?

@@ -179,10 +179,6 @@ Allow DNS lookups for \fB\-addnode\fR, \fB\-seednode\fR and \fB\-connect\fR (def
Query for peer addresses via DNS lookup, if low on addresses (default: 1
unless \fB\-connect\fR used)
.HP
\fB\-enablebip61\fR
Copy link
Contributor

@ryanofsky ryanofsky Oct 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these man files get generated from -help, so there's no need to make the same updates repeatedly

Copy link
Member Author

@MarcoFalke MarcoFalke Oct 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think generally it is fine to update them either in the same commit, or wait for the script to bump them later.

Here, I made the changes directly, so that git grep enablebip61 doesn't return these.

Copy link
Member

@laanwj laanwj Oct 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it's okay to do this (but also to not do this, for ex. because it can cause easily avoidable merge conflicts, as these are would-be hotspots), similar for updating the bitcoin_en translation

laanwj added a commit that referenced this issue Oct 9, 2019
fa25f43 p2p: Remove BIP61 reject messages (MarcoFalke)

Pull request description:

  Reject messages (BIP 61) appear in the following settings:

  * Parsing of reject messages (in case `-debug=net` is set, off by default). This has only been used for a single `LogPrint` call for several releases now. Such logging is completely meaningless to us and should thus be removed.

  * The sending of reject messages (in case `-enablebip61` is set, off by default). This can be used to debug a node that is under our control. Instead of hacking this debugging into the p2p protocol, it could be more easily achieved by parsing the debug log. (Use `-printtoconsole` to have it as stream, or read from the `debug.log` file like our python function `assert_debug_log` in the test framework does)

  Having to maintain all of this logic and code to accommodate debugging, which can be achieved by other means a lot easier, is a burden. It makes review on net processing changes a lot harder, since the reject message logic has to be carried around without introducing any errors or DOS vectors.

ACKs for top commit:
  jnewbery:
    utACK fa25f43
  laanwj:
    I'm still not 100% convinced that I like getting rid of BIP61 conceptually, but apparently everyone wants it, code review ACK fa25f43.
  ryanofsky:
    Code review ACK fa25f43

Tree-SHA512: daf55254202925e56be3d6cfb3c1c804e7a82cecb1dd1e5bd7b472bae989fd68ac4f21ec53fc46751353056fd645f7f877bebcb0b40920257991423a3d99e0be
@laanwj laanwj merged commit fa25f43 into bitcoin:master Oct 9, 2019
2 checks passed
@MarcoFalke MarcoFalke deleted the Mf1809-noBip61 branch Oct 9, 2019
sidhujag added a commit to syscoin/syscoin that referenced this issue Oct 9, 2019
fa25f43 p2p: Remove BIP61 reject messages (MarcoFalke)

Pull request description:

  Reject messages (BIP 61) appear in the following settings:

  * Parsing of reject messages (in case `-debug=net` is set, off by default). This has only been used for a single `LogPrint` call for several releases now. Such logging is completely meaningless to us and should thus be removed.

  * The sending of reject messages (in case `-enablebip61` is set, off by default). This can be used to debug a node that is under our control. Instead of hacking this debugging into the p2p protocol, it could be more easily achieved by parsing the debug log. (Use `-printtoconsole` to have it as stream, or read from the `debug.log` file like our python function `assert_debug_log` in the test framework does)

  Having to maintain all of this logic and code to accommodate debugging, which can be achieved by other means a lot easier, is a burden. It makes review on net processing changes a lot harder, since the reject message logic has to be carried around without introducing any errors or DOS vectors.

ACKs for top commit:
  jnewbery:
    utACK fa25f43
  laanwj:
    I'm still not 100% convinced that I like getting rid of BIP61 conceptually, but apparently everyone wants it, code review ACK fa25f43.
  ryanofsky:
    Code review ACK fa25f43

Tree-SHA512: daf55254202925e56be3d6cfb3c1c804e7a82cecb1dd1e5bd7b472bae989fd68ac4f21ec53fc46751353056fd645f7f877bebcb0b40920257991423a3d99e0be
@jnewbery
Copy link
Member

@jnewbery jnewbery commented Oct 9, 2019

We should announce that this has been merged on the bitcoin-dev mailing list, to close the https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016701.html thread.

@MarcoFalke
Copy link
Member Author

@MarcoFalke MarcoFalke commented Oct 16, 2019

MarcoFalke added a commit to MarcoFalke/bitcoin-core that referenced this issue Apr 12, 2020
b1b0cfe test: Remove REJECT message code (Hennadii Stepanov)

Pull request description:

  We no longer use REJECT p2p message:
  - bitcoin#15437
  - bitcoin#17004

ACKs for top commit:
  theStack:
    ACK bitcoin@b1b0cfe (nice dead code find)

Tree-SHA512: 0a662b282e921c3991aeb15f54d077837f1ef20bc2e3b0b35117bb97a21d1bd1c3e21458e5c18ba0ca02030d559e3e8e74dbd3d3e2b46dbe7bede550948c3b55
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Oct 20, 2020
Summary:
>Reject messages (BIP 61) appear in the following settings:
>   - Parsing of reject messages (in case -debug=net is set, off by default). This has only been used for a single LogPrint call for several releases now. Such logging is completely meaningless to us and should thus be removed.
>   - The sending of reject messages (in case -enablebip61 is set, off by default). This can be used to debug a node that is under our control. Instead of hacking this debugging into the p2p protocol, it could be more easily achieved by parsing the debug log. (Use -printtoconsole to have it as stream, or read from the debug.log file like our python function assert_debug_log in the test framework does)
>
> Having to maintain all of this logic and code to accommodate debugging, which can be achieved by other means a lot easier, is a burden. It makes review on net processing changes a lot harder, since the reject message logic has to be carried around without introducing any errors or DOS vectors.

Backport of Core [[bitcoin/bitcoin#15437 | PR15437]]

Test Plan: `ninja && ninja check-all`

Reviewers: O1 Bitcoin ABC, #bitcoin_abc, deadalnix

Reviewed By: O1 Bitcoin ABC, #bitcoin_abc, deadalnix

Subscribers: deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D7981
random-zebra added a commit to PIVX-Project/PIVX that referenced this issue Jul 25, 2021
…ic + BIP61 removal.

8e9f2bc Net_processing: Minimize tier two messages search time and improve code. (furszy)
56a78fc Clean few compiler warnings in blockassembler, validation, rpcwallet and evo_deterministicmns_tests files. (furszy)
c398797 Refactor: Name GetDataMsg enum and replace ppszTypeName array for direct GetCommand call. (furszy)
f99c5ed Move-only: IsTransactionInChain functions moved inside zpivchain.cpp and made static as them are only used there. (furszy)
2e1af2b Remove IsBlockHashInChain function. Only used inside IsTransactionInChain. (furszy)
38d49ea Document 'DisconnectOldProtocol' call rationale in block processing. (furszy)
4f62c04 Remove now unneeded `fAccepted` flag from ProcessNewBlock. (furszy)
0d4ba29 Move block spam filter check to BlockChecked (furszy)
bd85440 Refactor: get rid off the error prone CValidationState argument in ProcessNewBlock. (furszy)
22b1757 validation: Proper validation state set for "out of order" check in CheckBlock. (furszy)
9606474 net_processing: Clean CheckBlockSpam code and add proper cs_main locks in State() call and mapBlockIndex access. (furszy)
e6d2bd6 net_processing: Remove duplicate, and wrong, peer misbehaving due an invalid arriving block. (furszy)
6a5df01 p2p: Remove BIP61 reject messages (furszy)

Pull request description:

  Continuation of #2463. Another error prone topic.

  Essentially focused in the following points:
  * Remove duplicate and incorrect peer misbehaving:
  -1) The peer misbehaving score set for an invalid arriving block is being performed inside `PeerLogicValidation::BlockChecked`.
  -2) The `CValidationState` param in `ProcessNewBlock` does not return the error nor the invalidity reason if the block is marked invalid during the activate best chain process (block connection), an empty `CValidationState` is set. The `BlockChecked` signal is used instead to notify the block invalidity reason.

  * Get rid off `CValidationState` param in `ProcessNewBlock` and use only `BlockChecked` (own version of ae22357)
  * Removal of BIP61 (bitcoin#15437 and bitcoin#18609).
  -- BIP61 could be disabled first by default before its removal like upstream did (or moved to its own standalone PR), but.. i don't think that worth to continue maintaining it, most likely no one is using it and if someone is actually using it, isn't good to have any piece of software depending on it. --
  * Move block spam filter check to `BlockChecked`. Following the same peer-processing-logic / block-connection-logic division. Improves the spam protection as now blocks rejected during the connection process are going to pass through the spam filter check as well (before, only blocks rejected during `AcceptBlock` were passing through it).
  Plus, this improvement let me do some further cleanup and remove the extra `fAccepted` flag from `ProcessNewBlock`.
  * Create `GetDataMsg` enum and replace `ppszTypeName` array for direct `GetCommand` call.
  The array was duplicated with the `NetMsgType` constants

ACKs for top commit:
  random-zebra:
    ACK 8e9f2bc

Tree-SHA512: 362219e995c857ac9ad45119d0cad24b1afcb94c807d5af60d7008dd985c3b324b5f2938d60890f89d610d214f69f6a6d5ef12d2ce4e6016480ad984d00706c4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants