P2P: Mempool tracks locally submitted transactions to improve wallet privacy

[amitiuttarwar]

This PR introduces mempool tracking of unbroadcast transactions and periodic reattempts at initial broadcast. This is a part of the rebroadcast project, and a standalone privacy win.

The current rebroadcast logic is terrible for privacy because 1. only the source wallet rebroadcasts transactions and 2. it does so quite frequently. In the current system, if a user submits a transaction that does not immediately get broadcast to the network (eg. they are offline), this "rebroadcast" behavior is the safety net that can actually serve as the initial broadcast. So, keeping the attempts frequent is important for initial delivery within a reasonable timespan.

This PR aims to improve # 2 by reducing the wallet rebroadcast frequency to ~1/day from ~1/15 min. It achieves this by separating the notion of initial broadcast from rebroadcasts. With these changes, the mempool tracks locally submitted transactions & periodically reattempts initial broadcast. Transactions submitted via the wallet or RPC are added to an "unbroadcast" set & are removed when a peer sends a getdata request, or the transaction is removed from the mempool. Every 10-15 minutes, the node reattempts an initial broadcast. This enables reducing the wallet rebroadcast frequency while ensuring the transactions will be propagated to the network.

For privacy improvements around # 1, please see #16698.
Thank you to gmaxwell for the idea of how to break out this subset of functionality (#16698 (comment))

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• [doc / test / mempool] unbroadcast follow-ups #18807 ([WIP] unbroadcast follow-ups by amitiuttarwar)
• Add templated GetRandDuration<> #18781 (Add templated GetRandDuration<> by MarcoFalke)
• refactor: test: replace inv type magic numbers by constants #18764 (refactor: test: replace inv type magic numbers by constants by theStack)
• test: Add test for wtxid transaction relay #18446 (test: Add test for wtxid transaction relay by fjahr)
• Erlay: bandwidth-efficient transaction relay protocol #18261 (Erlay: bandwidth-efficient transaction relay protocol by naumenkogs)
• Use wtxid for transaction relay #18044 (Use wtxid for transaction relay by sdaftuar)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[amitiuttarwar]

travis failure seems unrelated- feature_fee_estimation.py

[maflcko]

It could be a crash, because node1 stops to log anything at all

[maflcko]

Concept ACK. Some design comments on the first commit 66ec557

[naumenkogs]

Concept ACK, reviewed code and left comments. Will test once we resolve the discussions.

[maflcko]

Some comments for the second commit

[ariard]

If I understand well this PR, it introduces a initial-broadcast reattempt mechanism through mempool tracking and GETDATA monitoring. Compare to wallet rebroadcast, the safety catch of new one isn't confirmation-based but network-based so once you have reasonable propagation (reasonable=1) you stop initial-broadcast. But I don't grasp where the privacy win is in itself, I would say that's a reliability improvement against network unknowns (maybe even a slight worsening because now a spying node only have to be connected to you for REATTEMPT_BROADCAST_INTERVAL instead of ~1/30min, which decrease attacker deployment cost ?)

META: can you keep your commit message line to 74 characters :) ?

[amitiuttarwar]

just pushed some changes:

• added functionality to persist unbroadcast set across restarts using mempool.dat & added some tests
• updates to locking when accessing unbroadcast set
• addressed misc review comments

I'm looking for review / feedback, particularly in the following areas:

1. that my locking logic makes sense

2. preference of handling the mempool.dat update.
   this current implementation doesn't update the mempool.dat version. Instead LoadMempool checks if we are at the end of the file. If not, reads the next line into the mempool unbroadcast set. this allows for compatibility with any current mempool.dat formats.

if we bump the version, it may cause some friction when users upgrade, but that might be negligible, and I don't expect persisting the unbroadcast set to be something that's super frequently used, especially in current network conditions.
then, we could have DumpMempool add a bool value that indicates whether or not there are unbroadcast txids, and LoadMempool could use a true to read the next line into the unbroadcast set.

I implemented the first way because I thought version interoperability would be simpler, but after seeing this implementation (and how annoyingly difficult it is to identify "are we at the end of the file?"), I am leaning towards bumping the version. But since I have it built out, thought I'd post the current version.

[amitiuttarwar]

looks like I have severely aggravated Travis. whoops 🙈. looking into it.