New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce a maximum size for locators. #13907
Conversation
Evoskuil is of the belief that some bitcoinj nodes were sending locators of size 100. If so, I'll probably suggest that we just increase this number to 100 to accommodate them rather than delay putting in a limit just to get a somewhat tighter one. I'd be thankful if some people with lots of inbound connections could run this with net logging and report back if anyone is getting disconnected and with what sizes. |
@@ -2018,6 +2018,12 @@ bool static ProcessMessage(CNode* pfrom, const std::string& strCommand, CDataStr | |||
uint256 hashStop; | |||
vRecv >> locator >> hashStop; | |||
|
|||
if (locator.vHave.size() > MAX_LOCATOR_SZ) { | |||
LogPrint(BCLog::NET, "getblocks locator size %lld > %d, disconnect peer=%d\n", locator.vHave.size(), MAX_LOCATOR_SZ, pfrom->GetId()); | |||
pfrom->fDisconnect = true; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a specific reason why you use fDisconnect
here, rather than Misbehaving
as in many other places around here? If there is, perhaps adding a comment would be helpful to future readers of the code (it would certainly be useful to me right now).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because I don't see any particular reason to ban the peer. We only need to disconnect rather than just ignoring the message because the peer might be counting on our response and ignoring it could make it get stuck when otherwise it might make progress with someone else.
We similarly disconnect for some messges when OutboundTargetReached, when we're limited and someone requests a block before the limited horizon, when the peer requests BIP37 services that we're not offering, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense, thanks for the clarification!
utACK, looks like a good (and straightforward) precaution to me |
@@ -2018,6 +2018,12 @@ bool static ProcessMessage(CNode* pfrom, const std::string& strCommand, CDataStr | |||
uint256 hashStop; | |||
vRecv >> locator >> hashStop; | |||
|
|||
if (locator.vHave.size() > MAX_LOCATOR_SZ) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that your BIP says: "A locator included in a getblock or getheaders message may include no more
than 64 hashes, including the final hash_stop hash", so unless I am mistaken, the >
should be replaced with a >=
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, if we end up following the BIP. I'm giving it about equal odds that we need to use a threshold higher than the BIP for now, after we see whats showing up on the network.
utACK 65d92ad beside my question if this should be implemented according to the BIP draft at https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-August/016285.html |
Seems reasonable. Tested ACK 65d92ad (just tested the getheaders logic). Functional test here: https://github.com/jnewbery/bitcoin/tree/pr13907.1 (commit jnewbery@377b1a6) |
@jnewbery FYI I have also written a test this morning, that additionally covers the case of blocks as well as a positive and negative test outcome: 917fb38397 |
@MarcoFalke - your test looks great. I've left a few comments on there. Once those are addressed, @gmaxwell should add your commit to this PR. |
The largest sensible size for a locator is log in the number of blocks. But, as noted by Coinr8d on BCT a maximum size message could encode a hundred thousand locators. If height were used to limit the messages that could open new attacks where peers on long low diff forks would get disconnected and end up stuck. Ideally, nodes first first learn to limit the size of locators they send before limiting what would be processed, but common implementations back off with an exponent of 2 and have an implicit limit of 2^32 blocks, so they already cannot produce locators over some size. This sets the limit to an absurdly high amount of 101 in order to maximize compatibility with existing software.
Sipa's observations show that there is software using 101, so I've updated to that size. The larger locators are stupid and pointless, but the harm in permitting them is negligible. |
utACK e254ff5 |
Note to reviewers: This pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
re-utACK e254ff5 (Only change is 64->101) |
utACK |
utACK e254ff5 |
e254ff5 Introduce a maximum size for locators. (Gregory Maxwell) Pull request description: The largest sensible size for a locator is log in the number of blocks. But, as noted by Coinr8d on BCT a maximum size message could encode a hundred thousand locators. If height were used to limit the messages that could open new attacks where peers on long low diff forks would get disconnected and end up stuck. Ideally, nodes first first learn to limit the size of locators they send before limiting what would be processed, but common implementations back off with an exponent of 2 and have an implicit limit of 2^32 blocks, so they already cannot produce locators over some size. Locators are cheap to process so allowing a few more is harmless, so this sets the maximum to 64-- which is enough for blockchains with 2^64 blocks before the get overhead starts increasing. Tree-SHA512: da28df9c46c988980da861046c62e6e7f93d0eaab3083d32e408d1062f45c00316d5e1754127e808c1feb424fa8e00e5a91aea2cc3b80326b71c148696f7cdb3
Actually the maximum number of supplied hashes for the current network height could exceed 190 as the algorithm produces 10*log2(nChainHeight), as it has been discussed earlier in btctalk thread. Plus, to avoid playing with this constant every few months, a safe_threshold has been suggested to be added, for the main chain height to become two times larger, like a decade later, this number would be 200. So, I suggest 200 for this constant, given we are ok to revise this number every 1-2 years. |
Above 10*log2(n) is an upper bound. The algorithm generates less hashes for smaller heights, e.g. for a chain with 10 blocks we get 10 hashes instead of 34 but for 1,000,000 blocks it is 160 which is much closer to 200. Any way, for current height we need 150 hashes in block locator to represent the active chain. |
It does not produce 10*log2(n), https://github.com/bitcoin/bitcoin/blob/master/src/chain.cpp#L30. Cheers. |
I just checked. Right! It is 10+log2(n), Now 101 seems to be TOO large, |
@AliAshrafD 101 was chosen here because we observed other implementations producing such large locators, and there is little reason to break compatibility. |
@sipa Although I'v not checked how bitcoinj reaches 100, no matter 40 or 101 or even 200 as long we are talking about locator issue. |
@AliAshrafD That's a very interesting theoretical question, but it seems completely off topic here. |
@sipa Well not completely off topic. By setting MAX_LOCATOR_SZ = 101, we are putting a cap on the maximum block height to be 2^90 forever, Aren't we? Still, I admit it is good enough for the problem of interest ;) just asking for other max-chain-height related issues if any. |
@AliAshrafD No, this is not a consensus rule, this is only relay. The limit can be changed in the future with some smart behavior as to know how large of locators a node can support (a new protocol version would probably be needed in that case). Perhaps there should be some check to make sure the maximum locator size is not exceeded. If it is, more message can just be sent. |
Also, 290 blocks is enough for 1.8 trillion times the age of the universe. I think we'll be fine. |
@ahow101 , @sipa I understand the block locator problem is solved, just curious about other max-chain-height related issues. I just published my work regarding this issue on btctalk, you are welcome to make any comments there. Thank you. |
Locking this since it's wandered off. Thanks to all for your comments! |
The largest sensible size for a locator is log in the number of blocks.
But, as noted by Coinr8d on BCT a maximum size message could encode a
hundred thousand locators. If height were used to limit the messages
that could open new attacks where peers on long low diff forks would
get disconnected and end up stuck.
Ideally, nodes first first learn to limit the size of locators they
send before limiting what would be processed, but common implementations
back off with an exponent of 2 and have an implicit limit of 2^32
blocks, so they already cannot produce locators over some size.
Locators are cheap to process so allowing a few more is harmless,
so this sets the maximum to 64-- which is enough for blockchains
with 2^64 blocks before the get overhead starts increasing.