Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mempool: rework rebroadcast logic to improve privacy #16698

Closed
wants to merge 7 commits into from

Conversation

@amitiuttarwar
Copy link
Member

@amitiuttarwar amitiuttarwar commented Aug 23, 2019

The current rebroadcast logic is terrible for privacy because only the source wallet will rebroadcast transactions, and does so quite frequently. This PR aims to improve privacy dramatically while preserving the benefits of rebroadcasting that ensure txns are successfully propagated through the network.

This PR introduces changes so nodes will resend transactions that it believes should have already been mined. It extracts the logic from the wallet into the mempool, so nodes will rebroadcast txns regardless of the originating wallet. Txns are defined as "should have been mined" by using the block assembler logic, and excluding txns that were recently added to the mempool. The wallet will resubmit txns to the mempool on a regular cadence to ensure those txns aren't dropped (due to eviction, expiry, etc..) before being confirmed.

For more information, see: https://gist.github.com/amitiuttarwar/b592ee410e1f02ac0d44fcbed4621dba

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Aug 23, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #17805 (test: Add test for rpcwhitelistdefault by emilengler)
  • #17786 (refactor: Nuke policy/fees->mempool circular dependencies by hebasto)
  • #17693 (rpc: Add generateblock to mine a custom set of transactions by andrewtoth)
  • #17484 (wallet: add cached m_is_ibd to remove Chain::isInitialBlockDownload by ariard)
  • #17303 (p2p: Stop relaying non-mempool txs, improve tx privacy slightly by MarcoFalke)
  • #17261 (Make ScriptPubKeyMan an actual interface and the wallet to have multiple by achow101)
  • #16910 (wallet: reduce loading time by using unordered maps by achow101)
  • #16528 ([WIP] Native Descriptor Wallets (take 2) by achow101)
  • #15719 (Drop Chain::requestMempoolTransactions method by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Loading

@amitiuttarwar amitiuttarwar force-pushed the rebroadcast branch 3 times, most recently from 13fd122 to c383bde Aug 23, 2019
@amitiuttarwar
Copy link
Member Author

@amitiuttarwar amitiuttarwar commented Aug 23, 2019

This code is ready for initial review.

there are still some to dos before it would be ready for merge:

  • identify expected rebroadcast traffic & worst case bandwidth usage.
  • persist setUnbroadcastTxIDs to mempool.dat
  • add functionality to run an automated job to cache min fee rate for txns to be included in block, then apply that filter to exclude txns with fee rate < min from rebroadcast set. this will reduce rebroadcast noise in scenarios where the mempool is emptying out.

there are also some follow-ups that can be addressed in separate PRs:

  • m_best_block_time is no longer used & can be removed & the wallet no longer needs to subscribe to UpdatedBlockTip() validation interface method
  • functionality to mark a peer (as "local" or such) so the mempool would still enforce initial broadcast for transactions received from one of these peers.

Loading

@jnewbery
Copy link
Member

@jnewbery jnewbery commented Aug 23, 2019

Concept ACK

Loading

Copy link
Member

@MarcoFalke MarcoFalke left a comment

Concept ACK. Some questions about why microseconds are truncated to seconds among other stuff.

Loading

src/net.h Outdated Show resolved Hide resolved
Loading
src/net.h Outdated Show resolved Hide resolved
Loading
src/miner.h Outdated Show resolved Hide resolved
Loading
src/txmempool.cpp Outdated Show resolved Hide resolved
Loading
Copy link
Contributor

@mzumsande mzumsande left a comment

Txns are defined as "should have been mined" by using the block assembler logic

I don’t understand 1) the concept “should have been mined”, and 2) how the block assembler logic achieves this.
As to 1) do you mean the txns should have been mined in a specific block/range of blocks, but weren’t? Should no rebroadcasts happen in an ideal world where miners have an identical mempool to ours and mine rationally?
As to 2) from what I understand, BlockAssembler creates a block template with 3/4*MAX_BLOCK_WEIGHT including the top feerate packages of our current mempool. Wouldn’t it always fill this block template with txns if our mempool is large enough, and therefore rather include 75% of the txns that we expect to be mined in the next block, instead of txns that should have been mined in the past?

Loading

src/net_processing.cpp Outdated Show resolved Hide resolved
Loading
test/functional/wallet_resendwallettransactions.py Outdated Show resolved Hide resolved
Loading
@amitiuttarwar
Copy link
Member Author

@amitiuttarwar amitiuttarwar commented Aug 26, 2019

thanks for the review @mzumsande !

I don’t understand 1) the concept “should have been mined" [...] do you mean the txns should have been mined in a specific block/range of blocks, but weren’t? Should no rebroadcasts happen in an ideal world where miners have an identical mempool to ours and mine rationally?

based on the local mempool, we are attempting to answer the question of what txns we think should have been mined. And saying if it wasn't, maybe there was an issue with relay.
You are correct- in a world where all mempools are consistent there wouldn't be rebroadcasts.

As to 2) from what I understand, BlockAssembler creates a block template with 3/4*MAX_BLOCK_WEIGHT including the top feerate packages of our current mempool. Wouldn’t it always fill this block template with txns if our mempool is large enough, and therefore rather include 75% of the txns that we expect to be mined in the next block, instead of txns that should have been mined in the past?

yes. you will start with txns you expect to be mined in the next block. the recency filter will (likely) remove some of those transactions. however, in the case of a mempool thats emptying out, the recency filter might not have much effect. for that I have this todo before the PR would be ready for merge:

add functionality to run an automated job to cache min fee rate for txns to be included in block, then apply that filter to exclude txns with fee rate < min from rebroadcast set

Loading

@mzumsande
Copy link
Contributor

@mzumsande mzumsande commented Aug 26, 2019

based on the local mempool, we are attempting to answer the question of what txns we think should have been mined.

What confuses me is how we can answer that without actually looking into recent blocks.
Considering that txns are removed from the mempool once they are included in a valid block, it is possible that the previous blocks removed the respective top of our previous versions of the mempool, so we are left with txns that were not at the top of the mempool earlier but are now, which we wouldn't want to rebroadcast.
Or the miners could have left out several high-fee-rate txns in favor of lower ones, in which case the higher ones are still present in our mempool and we would like to rebroadcast them.
Or there just might have been no new blocks found in the hour since the last rebroadcast, in which case we wouldn't need to rebroadcast.

How could we distinguish between these cases by just looking at our current mempool?

Loading

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Jan 30, 2020

Needs rebase

Loading

MarcoFalke added a commit that referenced this issue Feb 18, 2020
8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar)
7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar)
930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar)
1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar)
a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar)

Pull request description:

  This PR is to support functional tests by allowing the scheduler to be mocked via the RPC.

  It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner.

  This is currently used to support functional testing of the "unbroadcast" set tracking in #18038. If this patch is accepted, it would also be useful to simplify the code in #16698.

ACKs for top commit:
  MarcoFalke:
    ACK 8bca30e, only change is some style fixups 🕓

Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b
sidhujag added a commit to syscoin/syscoin that referenced this issue Feb 18, 2020
8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar)
7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar)
930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar)
1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar)
a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar)

Pull request description:

  This PR is to support functional tests by allowing the scheduler to be mocked via the RPC.

  It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner.

  This is currently used to support functional testing of the "unbroadcast" set tracking in bitcoin#18038. If this patch is accepted, it would also be useful to simplify the code in bitcoin#16698.

ACKs for top commit:
  MarcoFalke:
    ACK 8bca30e, only change is some style fixups 🕓

Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b
@MarcoFalke MarcoFalke removed this from the 0.20.0 milestone Mar 30, 2020
@amitiuttarwar
Copy link
Member Author

@amitiuttarwar amitiuttarwar commented Mar 31, 2020

closing this PR until #18038 gets merged.

if you're interested in moving this project forward, #18038 is ready for review :) thanks in advance!

Loading

codablock added a commit to codablock/dash that referenced this issue Apr 9, 2020
…kable time

1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5
  MarcoFalke:
    re-ACK 1a8f0d5
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
codablock added a commit to codablock/dash that referenced this issue Apr 12, 2020
…kable time

1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5
  MarcoFalke:
    re-ACK 1a8f0d5
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
codablock added a commit to codablock/dash that referenced this issue Apr 12, 2020
…kable time

1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5
  MarcoFalke:
    re-ACK 1a8f0d5
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
codablock added a commit to codablock/dash that referenced this issue Apr 12, 2020
…kable time

1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5
  MarcoFalke:
    re-ACK 1a8f0d5
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
codablock added a commit to codablock/dash that referenced this issue Apr 14, 2020
…kable time

1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5
  MarcoFalke:
    re-ACK 1a8f0d5
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
fanquake added a commit that referenced this issue Apr 29, 2020
…mprove wallet privacy

50fc4df [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eec [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)

Pull request description:

  This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.

  The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.

  This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.

  For privacy improvements around # 1, please see #16698.
  Thank you to gmaxwell for the idea of how to break out this subset of functionality (#16698 (comment))

ACKs for top commit:
  fjahr:
    Code review ACK 50fc4df
  MarcoFalke:
    ACK 50fc4df, I think this is ready for merge now 👻
  amitiuttarwar:
    The current tip `50fc4df` currently has 6 ACKs on it, so I've opened #18807 to address the last bits.
  jnewbery:
    utACK 50fc4df.
  ariard:
    Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
  robot-visions:
    ACK 50fc4df
  sipa:
    utACK 50fc4df
  naumenkogs:
    utACK 50fc4df

Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
sidhujag added a commit to syscoin/syscoin that referenced this issue Apr 29, 2020
…ns to improve wallet privacy

50fc4df [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eec [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)

Pull request description:

  This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.

  The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.

  This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.

  For privacy improvements around # 1, please see bitcoin#16698.
  Thank you to gmaxwell for the idea of how to break out this subset of functionality (bitcoin#16698 (comment))

ACKs for top commit:
  fjahr:
    Code review ACK 50fc4df
  MarcoFalke:
    ACK 50fc4df, I think this is ready for merge now 👻
  amitiuttarwar:
    The current tip `50fc4df` currently has 6 ACKs on it, so I've opened bitcoin#18807 to address the last bits.
  jnewbery:
    utACK 50fc4df.
  ariard:
    Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
  robot-visions:
    ACK 50fc4df
  sipa:
    utACK 50fc4df
  naumenkogs:
    utACK 50fc4df

Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
ComputerCraftr added a commit to ComputerCraftr/bitcoin that referenced this issue Jun 10, 2020
…ns to improve wallet privacy

50fc4df [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eec [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)

Pull request description:

  This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.

  The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.

  This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.

  For privacy improvements around # 1, please see bitcoin#16698.
  Thank you to gmaxwell for the idea of how to break out this subset of functionality (bitcoin#16698 (comment))

ACKs for top commit:
  fjahr:
    Code review ACK 50fc4df
  MarcoFalke:
    ACK 50fc4df, I think this is ready for merge now 👻
  amitiuttarwar:
    The current tip `50fc4df` currently has 6 ACKs on it, so I've opened bitcoin#18807 to address the last bits.
  jnewbery:
    utACK 50fc4df.
  ariard:
    Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
  robot-visions:
    ACK 50fc4df
  sipa:
    utACK 50fc4df
  naumenkogs:
    utACK 50fc4df

Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
ComputerCraftr added a commit to ComputerCraftr/bitcoin that referenced this issue Jun 10, 2020
…ns to improve wallet privacy

50fc4df [mempool] Persist unbroadcast set to mempool.dat (Amiti Uttarwar)
297a178 [test] Integration tests for unbroadcast functionality (Amiti Uttarwar)
6851502 [refactor/test] Extract P2PTxInvStore into test framework (Amiti Uttarwar)
dc1da48 [wallet] Update the rebroadcast frequency to be ~1/day. (Amiti Uttarwar)
e25e42f [p2p] Reattempt initial send of unbroadcast transactions (Amiti Uttarwar)
7e93eec [util] Add method that returns random time in milliseconds (Amiti Uttarwar)
89eeb4a [mempool] Track "unbroadcast" transactions (Amiti Uttarwar)

Pull request description:

  This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.

  The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.

  This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a `getdata` request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.

  For privacy improvements around # 1, please see bitcoin#16698.
  Thank you to gmaxwell for the idea of how to break out this subset of functionality (bitcoin#16698 (comment))

ACKs for top commit:
  fjahr:
    Code review ACK 50fc4df
  MarcoFalke:
    ACK 50fc4df, I think this is ready for merge now 👻
  amitiuttarwar:
    The current tip `50fc4df` currently has 6 ACKs on it, so I've opened bitcoin#18807 to address the last bits.
  jnewbery:
    utACK 50fc4df.
  ariard:
    Code Review ACK 50fc4df (minor points no need to invalid other ACKs)
  robot-visions:
    ACK 50fc4df
  sipa:
    utACK 50fc4df
  naumenkogs:
    utACK 50fc4df

Tree-SHA512: 2dd935d645d5e209f8abf87bfaa3ef0e4492705ce7e89ea64279cb27ffd37f4727fa94ad62d41be331177332f8edbebf3c7f4972f8cda10dd951b80a28ab3c0f
sidhujag added a commit to syscoin-core/syscoin that referenced this issue Nov 10, 2020
8bca30e [rpc] expose ability to mock scheduler via the rpc (Amiti Uttarwar)
7c8b6e5 [lib] add scheduler to node context (Amiti Uttarwar)
930d837 [test] add chainparams property to indicate chain allows time mocking (Amiti Uttarwar)
1cd43e8 [test] unit test for new MockForward scheduler method (Amiti Uttarwar)
a6f6359 [util] allow scheduler to be mocked (Amiti Uttarwar)

Pull request description:

  This PR is to support functional tests by allowing the scheduler to be mocked via the RPC.

  It adds a `MockForward` method to the scheduler class that iterates through the task queue and reschedules them to be `delta_seconds` sooner.

  This is currently used to support functional testing of the "unbroadcast" set tracking in bitcoin#18038. If this patch is accepted, it would also be useful to simplify the code in bitcoin#16698.

ACKs for top commit:
  MarcoFalke:
    ACK 8bca30e, only change is some style fixups 🕓

Tree-SHA512: 2a97fe8ade2b7fd1fb5cdfa1dcafb3227a377d7a847e3845a228bc119eb77824b4aefa43d922a06d583939b22725e223f308cf092961048079d36f6b1d9a639b
@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Jan 10, 2021

Is this work going to continue now that #18038 is done?

Loading

@amitiuttarwar
Copy link
Member Author

@amitiuttarwar amitiuttarwar commented Jan 12, 2021

@gmaxwell yes! I'm currently trying to work out a couple more kinks before continuing the review process on github & seeking feedback on approach.

Loading

Fabcien pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jan 21, 2021
Summary:
- Mempool tracks locally submitted transactions (wallet or rpc)
- Transactions are removed from set when the node receives a GETDATA request
  from a peer, or if the transaction is removed from the mempool.

PR description:
> This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.
>
> The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.
>
> This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a getdata request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.
>
> For privacy improvements around # 1, please see [[bitcoin/bitcoin#16698 | PR16698]].

This is a backport of Core [[bitcoin/bitcoin#18038 | PR18038]] [1/7]
bitcoin/bitcoin@89eeb4a

Test Plan: ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D9006
@amitiuttarwar
Copy link
Member Author

@amitiuttarwar amitiuttarwar commented Feb 8, 2021

this work is being continued in #21061 :)

Loading

ckti added a commit to ckti-ioncore-current/ion that referenced this issue Mar 28, 2021
…kable time

1a8f0d5 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de6303 [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5
  MarcoFalke:
    re-ACK 1a8f0d5
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
malbit added a commit to malbit/raptoreum that referenced this issue Nov 5, 2021
1a8f0d5a74d5cc0000456932babf35301f5c1686 [tools] update nNextInvSend to use mockable time (Amiti Uttarwar)
4de630354fc6808b9b13b9e82da1a82f2f50f26a [tools] add PoissonNextSend method that returns mockable time (Amiti Uttarwar)

Pull request description:

  Introduce a Poisson helper method that wraps the existing method to return `std::chrono::duration` type, which is mockable.

  Needed for bitcoin/bitcoin#16698.

ACKs for top commit:
  ajtowns:
    ACK 1a8f0d5a74d5cc0000456932babf35301f5c1686
  MarcoFalke:
    re-ACK 1a8f0d5a74d5cc0000456932babf35301f5c1686
  naumenkogs:
    ACK 1a8f0d5, and let's merge it and come back to it later.

Tree-SHA512: 7e2325d7c55fc0b4357cb86b83e0c218ba269f678c1786342d8bc380bfd9696373bc24ff124b9ff17a6e761c62b2b44ff5247c3911e2afdc7cc5c20417e8290b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Mempool improvements
Rebroadcasting
Linked issues

Successfully merging this pull request may close these issues.

None yet