Mempool: rework rebroadcast logic to improve privacy #16698

amitiuttarwar · 2019-08-23T16:48:58Z

The current rebroadcast logic is terrible for privacy because only the source wallet will rebroadcast transactions, and does so quite frequently. This PR aims to improve privacy dramatically while preserving the benefits of rebroadcasting that ensure txns are successfully propagated through the network.

This PR introduces changes so nodes will resend transactions that it believes should have already been mined. It extracts the logic from the wallet into the mempool, so nodes will rebroadcast txns regardless of the originating wallet. Txns are defined as "should have been mined" by using the block assembler logic, and excluding txns that were recently added to the mempool. The wallet will resubmit txns to the mempool on a regular cadence to ensure those txns aren't dropped (due to eviction, expiry, etc..) before being confirmed.

For more information, see: https://gist.github.com/amitiuttarwar/b592ee410e1f02ac0d44fcbed4621dba

DrahtBot · 2019-08-23T18:45:17Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#17805 (test: Add test for rpcwhitelistdefault by emilengler)
#17786 (refactor: Nuke policy/fees->mempool circular dependencies by hebasto)
#17693 (rpc: Add generateblock to mine a custom set of transactions by andrewtoth)
#17484 (wallet: add cached m_is_ibd to remove Chain::isInitialBlockDownload by ariard)
#17303 (p2p: Stop relaying non-mempool txs, improve tx privacy slightly by MarcoFalke)
#17261 (Make ScriptPubKeyMan an actual interface and the wallet to have multiple by achow101)
#16910 (wallet: reduce loading time by using unordered maps by achow101)
#16528 ([WIP] Native Descriptor Wallets (take 2) by achow101)
#15719 (Drop Chain::requestMempoolTransactions method by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

test/functional/wallet_resendwallettransactions.py

amitiuttarwar · 2019-08-23T22:23:04Z

This code is ready for initial review.

there are still some to dos before it would be ready for merge:

identify expected rebroadcast traffic & worst case bandwidth usage.
persist setUnbroadcastTxIDs to mempool.dat
add functionality to run an automated job to cache min fee rate for txns to be included in block, then apply that filter to exclude txns with fee rate < min from rebroadcast set. this will reduce rebroadcast noise in scenarios where the mempool is emptying out.

there are also some follow-ups that can be addressed in separate PRs:

m_best_block_time is no longer used & can be removed & the wallet no longer needs to subscribe to UpdatedBlockTip() validation interface method
functionality to mark a peer (as "local" or such) so the mempool would still enforce initial broadcast for transactions received from one of these peers.

jnewbery · 2019-08-23T22:24:32Z

Concept ACK

maflcko

Concept ACK. Some questions about why microseconds are truncated to seconds among other stuff.

src/net.h

src/miner.h

src/txmempool.cpp

mzumsande

Txns are defined as "should have been mined" by using the block assembler logic

I don’t understand 1) the concept “should have been mined”, and 2) how the block assembler logic achieves this.
As to 1) do you mean the txns should have been mined in a specific block/range of blocks, but weren’t? Should no rebroadcasts happen in an ideal world where miners have an identical mempool to ours and mine rationally?
As to 2) from what I understand, BlockAssembler creates a block template with 3/4*MAX_BLOCK_WEIGHT including the top feerate packages of our current mempool. Wouldn’t it always fill this block template with txns if our mempool is large enough, and therefore rather include 75% of the txns that we expect to be mined in the next block, instead of txns that should have been mined in the past?

src/net_processing.cpp

test/functional/wallet_resendwallettransactions.py

amitiuttarwar · 2019-08-26T22:20:52Z

thanks for the review @mzumsande !

I don’t understand 1) the concept “should have been mined" [...] do you mean the txns should have been mined in a specific block/range of blocks, but weren’t? Should no rebroadcasts happen in an ideal world where miners have an identical mempool to ours and mine rationally?

based on the local mempool, we are attempting to answer the question of what txns we think should have been mined. And saying if it wasn't, maybe there was an issue with relay.
You are correct- in a world where all mempools are consistent there wouldn't be rebroadcasts.

As to 2) from what I understand, BlockAssembler creates a block template with 3/4*MAX_BLOCK_WEIGHT including the top feerate packages of our current mempool. Wouldn’t it always fill this block template with txns if our mempool is large enough, and therefore rather include 75% of the txns that we expect to be mined in the next block, instead of txns that should have been mined in the past?

yes. you will start with txns you expect to be mined in the next block. the recency filter will (likely) remove some of those transactions. however, in the case of a mempool thats emptying out, the recency filter might not have much effect. for that I have this todo before the PR would be ready for merge:

add functionality to run an automated job to cache min fee rate for txns to be included in block, then apply that filter to exclude txns with fee rate < min from rebroadcast set

mzumsande · 2019-08-26T22:51:23Z

based on the local mempool, we are attempting to answer the question of what txns we think should have been mined.

What confuses me is how we can answer that without actually looking into recent blocks.
Considering that txns are removed from the mempool once they are included in a valid block, it is possible that the previous blocks removed the respective top of our previous versions of the mempool, so we are left with txns that were not at the top of the mempool earlier but are now, which we wouldn't want to rebroadcast.
Or the miners could have left out several high-fee-rate txns in favor of lower ones, in which case the higher ones are still present in our mempool and we would like to rebroadcast them.
Or there just might have been no new blocks found in the hour since the last rebroadcast, in which case we wouldn't need to rebroadcast.

How could we distinguish between these cases by just looking at our current mempool?

8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar) 7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar) 930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar) 1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar) a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar) Pull request description: This PR is to support functional tests by allowing the scheduler to be mocked via the RPC. It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner. This is currently used to support functional testing of the "unbroadcast" set tracking in #18038. If this patch is accepted, it would also be useful to simplify the code in #16698. ACKs for top commit: MarcoFalke: ACK 8bca30e, only change is some style fixups 🕓 Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b

8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar) 7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar) 930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar) 1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar) a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar) Pull request description: This PR is to support functional tests by allowing the scheduler to be mocked via the RPC. It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner. This is currently used to support functional testing of the "unbroadcast" set tracking in bitcoin#18038. If this patch is accepted, it would also be useful to simplify the code in bitcoin#16698. ACKs for top commit: MarcoFalke: ACK 8bca30e, only change is some style fixups 🕓 Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b

amitiuttarwar · 2020-03-31T00:25:03Z

closing this PR until #18038 gets merged.

if you're interested in moving this project forward, #18038 is ready for review :) thanks in advance!

…kable time 1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar) 4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar) Pull request description: Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable. Needed for bitcoin#16698. ACKs for top commit: ajtowns: ACK 1a8f0d5 MarcoFalke: re-ACK 1a8f0d5 naumenkogs: ACK 1a8f0d5, and let's merge it and come back to it later. Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b

…mprove wallet privacy 50fc4df [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar) 297a178 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar) 6851502 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar) dc1da48 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar) e25e42f [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar) 7e93eec [util] Add method that returns random time in milliseconds (Amiti Uttarwar) 89eeb4a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar) Pull request description: This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win. The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan. This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network. For privacy improvements around # 1, please see #16698. Thank you to gmaxwell for the idea of how to break out this subset of functionality (#16698 (comment)) ACKs for top commit: fjahr: Code review ACK 50fc4df MarcoFalke: ACK 50fc4df, I think this is ready for merge now 👻 amitiuttarwar: The current tip `50fc4df` currently has 6 ACKs on it, so I've opened #18807 to address the last bits. jnewbery: utACK 50fc4df. ariard: Code Review ACK 50fc4df (minor points no need to invalid other ACKs) robot-visions: ACK 50fc4df sipa: utACK 50fc4df naumenkogs: utACK 50fc4df Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f

…ns to improve wallet privacy 50fc4df [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar) 297a178 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar) 6851502 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar) dc1da48 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar) e25e42f [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar) 7e93eec [util] Add method that returns random time in milliseconds (Amiti Uttarwar) 89eeb4a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar) Pull request description: This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win. The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan. This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network. For privacy improvements around # 1, please see bitcoin#16698. Thank you to gmaxwell for the idea of how to break out this subset of functionality (bitcoin#16698 (comment)) ACKs for top commit: fjahr: Code review ACK 50fc4df MarcoFalke: ACK 50fc4df, I think this is ready for merge now 👻 amitiuttarwar: The current tip `50fc4df` currently has 6 ACKs on it, so I've opened bitcoin#18807 to address the last bits. jnewbery: utACK 50fc4df. ariard: Code Review ACK 50fc4df (minor points no need to invalid other ACKs) robot-visions: ACK 50fc4df sipa: utACK 50fc4df naumenkogs: utACK 50fc4df Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f

8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar) 7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar) 930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar) 1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar) a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar) Pull request description: This PR is to support functional tests by allowing the scheduler to be mocked via the RPC. It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner. This is currently used to support functional testing of the "unbroadcast" set tracking in bitcoin#18038. If this patch is accepted, it would also be useful to simplify the code in bitcoin#16698. ACKs for top commit: MarcoFalke: ACK 8bca30e, only change is some style fixups 🕓 Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b

gmaxwell · 2021-01-10T14:30:12Z

Is this work going to continue now that #18038 is done?

amitiuttarwar · 2021-01-12T02:34:33Z

@gmaxwell yes! I'm currently trying to work out a couple more kinks before continuing the review process on github & seeking feedback on approach.

Summary: - Mempool tracks locally submitted transactions (wallet or rpc) - Transactions are removed from set when the node receives a GETDATA request from a peer, or if the transaction is removed from the mempool. PR description: > This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win. > > The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan. > > This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a getdata request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network. > > For privacy improvements around # 1, please see [[bitcoin/bitcoin#16698 | PR16698]]. This is a backport of Core [[bitcoin/bitcoin#18038 | PR18038]] [1/7] bitcoin/bitcoin@89eeb4a Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Subscribers: majcosta Differential Revision: https://reviews.bitcoinabc.org/D9006

amitiuttarwar · 2021-02-08T17:55:15Z

this work is being continued in #21061 :)

1a8f0d5a74d5cc0000456932babf35301f5c1686 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar) 4de630354fc6808b9b13b9e82da1a82f2f50f26a [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar) Pull request description: Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable. Needed for bitcoin/bitcoin#16698. ACKs for top commit: ajtowns: ACK 1a8f0d5a74d5cc0000456932babf35301f5c1686 MarcoFalke: re-ACK 1a8f0d5a74d5cc0000456932babf35301f5c1686 naumenkogs: ACK 1a8f0d5, and let's merge it and come back to it later. Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b

…kable time 1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar) 4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar) Pull request description: Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable. Needed for bitcoin#16698. ACKs for top commit: ajtowns: ACK 1a8f0d5 MarcoFalke: re-ACK 1a8f0d5 naumenkogs: ACK 1a8f0d5, and let's merge it and come back to it later. Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b

8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar) 7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar) 930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar) 1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar) a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar) Pull request description: This PR is to support functional tests by allowing the scheduler to be mocked via the RPC. It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner. This is currently used to support functional testing of the "unbroadcast" set tracking in bitcoin#18038. If this patch is accepted, it would also be useful to simplify the code in bitcoin#16698. ACKs for top commit: MarcoFalke: ACK 8bca30e, only change is some style fixups 🕓 Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b

DrahtBot added Mempool Mining P2P RPC/REST/ZMQ Tests TX fees and policy Wallet labels Aug 23, 2019

amitiuttarwar force-pushed the rebroadcast branch 3 times, most recently from 13fd122 to c383bde Compare August 23, 2019 20:27

maflcko reviewed Aug 23, 2019

View reviewed changes

test/functional/wallet_resendwallettransactions.py Outdated Show resolved Hide resolved

amitiuttarwar force-pushed the rebroadcast branch from c383bde to ba33693 Compare August 23, 2019 21:56

amitiuttarwar force-pushed the rebroadcast branch from ba33693 to b1d0a54 Compare August 23, 2019 22:35

meshcollider added Wallet and removed Mining RPC/REST/ZMQ TX fees and policy Tests Wallet P2P labels Aug 24, 2019

fanquake added the P2P label Aug 24, 2019

amitiuttarwar force-pushed the rebroadcast branch from b1d0a54 to 4bda142 Compare August 24, 2019 13:08

maflcko reviewed Aug 25, 2019

View reviewed changes

src/net.h Outdated Show resolved Hide resolved

src/net.h Outdated Show resolved Hide resolved

src/miner.h Outdated Show resolved Hide resolved

src/txmempool.cpp Outdated Show resolved Hide resolved

mzumsande reviewed Aug 26, 2019

View reviewed changes

src/net_processing.cpp Outdated Show resolved Hide resolved

test/functional/wallet_resendwallettransactions.py Outdated Show resolved Hide resolved

maflcko mentioned this pull request Aug 27, 2019

Future PRs bitcoin-core-review-club/website#14

Closed

This was referenced Jan 31, 2020

Util: Allow scheduler to be mocked #18037

Merged

P2P: Mempool tracks locally submitted transactions to improve wallet privacy #18038

Merged

kristovatlas mentioned this pull request Feb 11, 2020

add to threat model to cover this bitcoin core rebroadcast feature OpenBitcoinPrivacyProject/wallet-ratings#146

Open

maflcko removed this from the 0.20.0 milestone Mar 30, 2020

amitiuttarwar closed this Mar 31, 2020

ariard mentioned this pull request Aug 27, 2020

Transactions propagation design goals #19820

Closed

amitiuttarwar mentioned this pull request Feb 2, 2021

[p2p] Introduce node rebroadcast module #21061

Closed

JaredTate mentioned this pull request Oct 24, 2021

WIP: Feature/8.22.0 Random Crash Bug & Missing Dandelion Relay TX Code DigiByte-Core/digibyte#77

Closed

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mempool: rework rebroadcast logic to improve privacy #16698

Mempool: rework rebroadcast logic to improve privacy #16698

amitiuttarwar commented Aug 23, 2019

DrahtBot commented Aug 23, 2019 •

edited

amitiuttarwar commented Aug 23, 2019

jnewbery commented Aug 23, 2019

maflcko left a comment

mzumsande left a comment •

edited

amitiuttarwar commented Aug 26, 2019

mzumsande commented Aug 26, 2019

amitiuttarwar commented Mar 31, 2020

gmaxwell commented Jan 10, 2021

amitiuttarwar commented Jan 12, 2021

amitiuttarwar commented Feb 8, 2021

Mempool: rework rebroadcast logic to improve privacy #16698

Mempool: rework rebroadcast logic to improve privacy #16698

Conversation

amitiuttarwar commented Aug 23, 2019

DrahtBot commented Aug 23, 2019 • edited

Conflicts

amitiuttarwar commented Aug 23, 2019

jnewbery commented Aug 23, 2019

maflcko left a comment

Choose a reason for hiding this comment

mzumsande left a comment • edited

Choose a reason for hiding this comment

amitiuttarwar commented Aug 26, 2019

mzumsande commented Aug 26, 2019

amitiuttarwar commented Mar 31, 2020

gmaxwell commented Jan 10, 2021

amitiuttarwar commented Jan 12, 2021

amitiuttarwar commented Feb 8, 2021

DrahtBot commented Aug 23, 2019 •

edited

mzumsande left a comment •

edited