Replace global trickle node with random delays #7125

sipa · 2015-11-28T14:56:58Z

This is based upon and partially replaces #5989, and rebased on top of #7100.

We used to have a trickle node, a node which was chosen in each iteration of the send loop that was privileged and allowed to send out queued up non-time critical messages. Since the removal of the fixed sleeps in the network code, this resulted in fast and attackable (see #7123) treatment of such broadcasts.

This pull request changes each of these trickle-dependant mechanisms with a per-node and per-feature random delay timer, improving privacy and performance by making queueing useful again.

The commits are mostly squashed/rebased versions taken from #5989 by @pstratem, except:

No randomization of tx invs (more code changes, and may not interact well with transactions that need to be sent after their dependencies).
No removal of the "1/4 blast to all" behaviour, as [WIP] Make trickle logic useful again, delay trickle when past upload limit. #7123 proposes an alternative.
No removal of the "announce yourself to outbound peers" behaviour.

sipa · 2015-11-29T12:21:37Z

Rebased on top of #7125, and make blocks not use the vInventoryToSend system anymore (making it safer to make it filter like #7100).

sipa · 2015-11-30T12:37:32Z

Rebased on top of #7133.

gmaxwell · 2015-12-05T00:25:36Z

Could use some rebasing love, as the bloomy stuff was merged.

sipa · 2015-12-05T06:25:04Z

Rebased.

laanwj · 2015-12-09T08:02:22Z

Some nodes connect to to as many peers as possible and flood them with pings, apparently to trigger the issue where pings interfere with trickle delays. So this is being actively exploited.

Added 0.12 milestone.

gmaxwell · 2015-12-10T11:00:42Z

I am currently testing this on top of 0.12.

morcos · 2015-12-10T16:27:57Z

utACK

sdaftuar · 2015-12-11T16:11:31Z

src/main.cpp

-        {
+
+        if (pto->nNextLocalAddrSend < nNow) {
+            AdvertizeLocal(pto);


Should we be guarding this with a !IsInitialBlockDownload() as before?

sipa · 2015-12-11T21:13:36Z

Addressed @sdaftuar's comment, rebased, and switched to Poisson-distributed sends, so nobody can ever make a guess about when the next send will happen.

We used to have a trickle node, a node which was chosen in each iteration of the send loop that was privileged and allowed to send out queued up non-time critical messages. Since the removal of the fixed sleeps in the network code, this resulted in fast and attackable treatment of such broadcasts. This pull request changes the 3 remaining trickle use cases by random delays: * Local address broadcast (while also removing the the wiping of the seen filter) * Address relay * Inv relay (for transactions; blocks are always relayed immediately) The code is based on older commits by Patrick Strateman.

gmaxwell · 2015-12-14T12:22:04Z

ACK

5400ef6 Replace trickle nodes with per-node/message Poisson delays (Pieter Wuille)

We used to have a trickle node, a node which was chosen in each iteration of the send loop that was privileged and allowed to send out queued up non-time critical messages. Since the removal of the fixed sleeps in the network code, this resulted in fast and attackable treatment of such broadcasts. This pull request changes the 3 remaining trickle use cases by random delays: * Local address broadcast (while also removing the the wiping of the seen filter) * Address relay * Inv relay (for transactions; blocks are always relayed immediately) The code is based on older commits by Patrick Strateman. Github-Pull: #7125 Rebased-From: 5400ef6

laanwj · 2016-02-04T08:07:12Z

This is cherry-picked to 0.12 as 10b88be

dgenr8 · 2018-03-17T00:06:02Z

Where was the discussion of the choice of 5 seconds as the mean delay between inventory broadcasts? This was 7-10x the median transaction propagation time of the entire network at that time.

6f41b3e [QA] Missing mempool sync in pos_coldStaking and zc_publicspends tests (random-zebra) efaf727 net: correctly initialize nMinPingUsecTime (Wladimir J. van der Laan) 61c8ffe Do not add random inbound peers to addrman. (Gregory Maxwell) e6a1726 Added feeler connections increasing good addrs in the tried table. (Ethan Heilman) 070b6fb Actually only use filterInventoryKnown with MSG_TX inventory messages. (Gregory Maxwell) 9ac6b28 Only use filterInventoryKnown with MSG_TX inventory messages. (Patick Strateman) 01273db Rename setInventoryKnown filterInventoryKnown (Patick Strateman) 4f11eb2 Remove mruset as it is no longer used. (Gregory Maxwell) 2e3b05c Replace setInventoryKnown with a rolling bloom filter. (Gregory Maxwell) 409aa83 Replace trickle nodes with per-node/message Poisson delays (Pieter Wuille) 93e8c46 Move recentRejects initialization to top of InitBlockIndex (Wladimir J. van der Laan) 9a59420 Keep track of recently rejected transactions (Peter Todd) 34ee777 Only use randomly created nonces in CRollingBloomFilter. (Pieter Wuille) 338d346 Make CRollingBloomFilter set nTweak for you (Peter Todd) dcd15bc Reuse vector hashing code for uint256 (Pieter Wuille) 3230143 Add uint256 support to CRollingBloomFilter (Peter Todd) 128d644 Better mruset unit test (Pieter Wuille) 89740ed Use ring buffer of set iterators instead of deque of copies in mruset (Pieter Wuille) 14c88ee Replace mruset setAddrKnown with CRollingBloomFilter addrKnown (Gavin Andresen) e0bebbd Rolling bloom filter class (Gavin Andresen) 7c03bd5 Add correct bool combiner for net signals (Pieter Wuille) 5cb5fd6 Stop exporting ConnectNode (Fuzzbawls) 819295d Stop using ConnectNode in layer 2 code (Fuzzbawls) 851b6b4 net: No need to export DumpBanlist (Cory Fields) 4486d4e net: make Ban/Unban/ClearBan functionality consistent (Cory Fields) a5e278d net: Drop CNodeRef for AttemptToEvictConnection (Cory Fields) 9fd357d net: use the exposed GetNodeSignals() rather than g_signals directly (Cory Fields) 7962bcc net: remove unused set (Cory Fields) fabf358 Use network group instead of CNetAddr in final pass to select node to disconnect (Patrick Strateman) 7f030fe Fix comment (Patrick Strateman) 18af800 Acquire cs_vNodes before changing refrence counts (Patrick Strateman) 7aa827f CNodeRef copy constructor and assignment operator (Patrick Strateman) b3f95e7 Return false early if vEvictionCandidates is empty (Patrick Strateman) 85886c9 Better support for nodes with non-standard nMaxConnections (Patrick Strateman) 9c9e55b RAII wrapper for CNode* (Patrick Strateman) e92780d Add comments to AttemptToEvictConnection (Patrick Strateman) 0ca7ce3 Prefer to disconnect peers in favor of whitelisted peers (Patrick Strateman) a1c4aaf AttemptToEvictConnection (Patrick Strateman) aa7ce9b Record nMinPingUsecTime (Patrick Strateman) fd7bab0 Refactor: Move failure conditions to the top of AcceptConnection (Patrick Strateman) fcb732b Refactor: Bail early in AcceptConnection (Patrick Strateman) 411766d Refactor: AcceptConnection (Patrick Strateman) Pull request description: This is a culmination of several upstream PRs touching the P2P/Networking code, and resulting in a state just prior to the P2P/Network encapsulation, which will be it's own PR. Backported upstream PRs included here: - bitcoin#5859 - bitcoin#6064 - bitcoin#6374 - bitcoin#6498 - bitcoin#6636 - bitcoin#7133 - bitcoin#7125 - bitcoin#7906 - bitcoin#8282 - bitcoin#8594 ACKs for top commit: random-zebra: Great job. ACK 6f41b3e and merging... furszy: Have been running it the past days and all is looking good, ACK 6f41b3e . Tree-SHA512: 1cc4b1271516f000a06141b5b069f3ee00f3eb77056e40d2c021c484f749d9d8db2b76ce490f63572372705b646fad342666f6f90ca5fc69abcacf7b207d058f

65273fa Clear m_addr_known before our periodic self-advertisement (Suhas Daftuar) Pull request description: We use a rolling bloom filter to track which addresses we've previously sent a peer, but after #7125 we no longer clear it every day before our own announcement. This looks to me like an oversight which has the effect of reducing the frequency with which we actually self-announce our own address, so this reintroduces resetting that filter. ACKs for top commit: naumenkogs: ACK 65273fa laanwj: Code review ACK 65273fa sipa: utACK 65273fa Tree-SHA512: 602c155fb6d2249b054fcb6f1c0dd17143605ceb87132286bbd90babf26d258ff6c41f9925482c17e2be41805d33f9b83926cb447f394969ffecd4bccfa0a64f

65273fa Clear m_addr_known before our periodic self-advertisement (Suhas Daftuar) Pull request description: We use a rolling bloom filter to track which addresses we've previously sent a peer, but after bitcoin#7125 we no longer clear it every day before our own announcement. This looks to me like an oversight which has the effect of reducing the frequency with which we actually self-announce our own address, so this reintroduces resetting that filter. ACKs for top commit: naumenkogs: ACK 65273fa laanwj: Code review ACK 65273fa sipa: utACK 65273fa Tree-SHA512: 602c155fb6d2249b054fcb6f1c0dd17143605ceb87132286bbd90babf26d258ff6c41f9925482c17e2be41805d33f9b83926cb447f394969ffecd4bccfa0a64f

ZIP 239 preparations 1 This is the first of several backports to prepare for ZIP 239. The primary change is altering `mapRelay` to store `CTransaction`s, which we need because ZIP 239 requires changing `Inv` messages based on transaction versions. The other changes are mainly for conflict removal but are also independently useful. Backports the following upstream PRs: - bitcoin/bitcoin#6889 - bitcoin/bitcoin#7125 - bitcoin/bitcoin#7862 - bitcoin/bitcoin#7877

sipa mentioned this pull request Nov 28, 2015

[WIP] Make trickle logic useful again, delay trickle when past upload limit. #7123

Closed

sipa force-pushed the timetrickle branch from 2317a58 to 4f819fa Compare November 28, 2015 15:08

sipa mentioned this pull request Nov 28, 2015

improve addr/inv trickle logic #5989

Closed

sipa force-pushed the timetrickle branch 3 times, most recently from caf4d27 to fb7f835 Compare November 29, 2015 12:20

sipa mentioned this pull request Nov 29, 2015

Replace setInventoryKnown with a rolling bloom filter. #7100

Closed

sipa force-pushed the timetrickle branch 2 times, most recently from 70061d5 to 448d663 Compare November 30, 2015 12:37

sipa force-pushed the timetrickle branch from 448d663 to a9fea8e Compare November 30, 2015 15:22

laanwj added the P2P label Dec 1, 2015

sipa mentioned this pull request Dec 3, 2015

Net: Remove "Address refresh broadcast" logic. #6745

Closed

sipa force-pushed the timetrickle branch from a9fea8e to 33a0ec2 Compare December 5, 2015 06:24

laanwj added the Needs backport label Dec 9, 2015

laanwj added this to the 0.12.0 milestone Dec 9, 2015

sdaftuar reviewed Dec 11, 2015
View reviewed changes

sipa force-pushed the timetrickle branch from 33a0ec2 to fca1db2 Compare December 11, 2015 21:12

sipa force-pushed the timetrickle branch from fca1db2 to 5400ef6 Compare December 11, 2015 21:22

laanwj merged commit 5400ef6 into bitcoin:master Dec 14, 2015

laanwj added a commit that referenced this pull request Dec 14, 2015

Merge pull request #7125

7a50401

5400ef6 Replace trickle nodes with per-node/message Poisson delays (Pieter Wuille)

ghost mentioned this pull request Jan 17, 2016

Improve 0-conf tracking bitcoinclassic/bitcoinclassic#16

Closed

laanwj removed the Needs backport label Feb 4, 2016

rebroad mentioned this pull request Aug 26, 2016

Provide bloom services to whitelisted nodes. #8587

Closed

dgenr8 mentioned this pull request May 18, 2017

Networking improvements bitcoinxt/bitcoinxt#203

Merged

str4d mentioned this pull request Jul 14, 2017

Bitcoin 0.12 P2P/Net PRs 1 zcash/zcash#2534

Draft

Fuzzbawls mentioned this pull request Jun 23, 2020

[Net] Pre-requirements for network encapsulation PIVX-Project/PIVX#1704

Merged

sdaftuar mentioned this pull request Dec 3, 2020

p2p: periodically clear m_addr_known #20561

Merged

jnewbery mentioned this pull request Jul 27, 2021

[p2p] Reduce addr blackholes #21528

Merged

str4d mentioned this pull request Aug 4, 2021

ZIP 239 preparations 1 zcash/zcash#5265

Merged

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace global trickle node with random delays #7125

Replace global trickle node with random delays #7125

sipa commented Nov 28, 2015

sipa commented Nov 29, 2015

sipa commented Nov 30, 2015

gmaxwell commented Dec 5, 2015

sipa commented Dec 5, 2015

laanwj commented Dec 9, 2015

gmaxwell commented Dec 10, 2015

morcos commented Dec 10, 2015

sdaftuar Dec 11, 2015

sipa commented Dec 11, 2015

gmaxwell commented Dec 14, 2015

laanwj commented Feb 4, 2016

dgenr8 commented Mar 17, 2018

Replace global trickle node with random delays #7125

Replace global trickle node with random delays #7125

Conversation

sipa commented Nov 28, 2015

sipa commented Nov 29, 2015

sipa commented Nov 30, 2015

gmaxwell commented Dec 5, 2015

sipa commented Dec 5, 2015

laanwj commented Dec 9, 2015

gmaxwell commented Dec 10, 2015

morcos commented Dec 10, 2015

sdaftuar Dec 11, 2015

Choose a reason for hiding this comment

sipa commented Dec 11, 2015

gmaxwell commented Dec 14, 2015

laanwj commented Feb 4, 2016

dgenr8 commented Mar 17, 2018