Discussion on the p2p network implementation

[Coinjoiner21]

While testing a fresh install of Wasabi over Tor, I noticed that it needed more than 20 minutes to download the 13k filters after the last checkpoint, with the process sitting at 600-800 MB of memory the whole time. That made me suspect that something is going wrong with the way the client talks to the p2p network, so I started digging into #14546 and the 17 PRs that improved it since then (#14574, #14580, #14606, #14616, #14669, #14698, #14772, #14788, #14851, #14975, #14986, #14995, #14996, #14999, #15003, #15004, #15108). I found quite a lot of issues, each one small on its own, that together make the sync slow and fragile.

The ultimate goal of this thread is to lay out the problems and use it as index for the upcoming PRs that solve the main issues.

How I measured

Everything below is from master (aef74b4), daemon only, mainnet, bundled Tor, fresh data directory, debug log level. In front of Tor I put a SOCKS relay that counts the bytes of every node's stream, so I know which node sent what and when. I ran 10 full syncs; the three below are representative. The payload a fresh install actually needs is about 298 MB: 219 MB of filters and 79 MB of block headers.

Run	Build	First node	Block headers done	All filters stored	Downloaded
A	master	125 sec	17 min	21.4 min	600 MB
A2	master	122 sec	12 min	18.8 min	655 MB
ALL	prototype of fixes 1 to 5	146 sec	147 sec	8.6 min	256 MB

The problems are all about how the client distributes the work between the nodes.

---

The issues, from most expensive to least

1. The client downloads the whole header chain from every node

NBitcoin's ChainBehavior is attached to every node, and each instance has its own cursor that starts at the shared tip at handshake time and walks forward 2000 headers at a time until it reaches the node's tip. Nothing stops the download from a node that started early when another node has already brought the chain to the tip. In run A the two nodes connected at 125 sec each sent 80 MB at 69 KB/s for 19 minutes; in A2 the header phase received 320 MB from 5 nodes, 4 times the chain. This phase is bounded by the circuit speed of whichever node happened to come first. Side costs: 973k ChainedBlock objects in memory (520 MB RSS right after start), a 109 MB BlockHeadersMain.dat written only on graceful exit, and header batches queued in front of everything else on that connection.

The fix in #15228: start the chain at the hardcoded header of the retarget boundary below the filter checkpoint (959616 for 960000). FilterCheckpoints already hardcodes the block hash at 960000, so the trust model does not change. Result: headers done 5 sec after the first node, 1 MB per node instead of 79 MB, 140-240 MB RSS.

2. A getcfheaders sent to a node that is streaming filters times out

A node that is streaming a 500-filter range (8.4 MB) is handed a filter header range a few seconds later, because the 2 sec tick gives a free header range to any node that has none. Core answers in order, so the 64 KB cfheaders reply sits behind the rest of the filters and cannot arrive within the 45 sec header timeout on a 95 KB/s circuit. The client then disconnects a healthy node and throws away the half-received filter range.

The fix in #15231: one request type in flight per node, headers first.

3. Filter ranges are handed out twice

FilterSynchronizationState re-assigns a range after 25 sec (headers) / 30 sec (filters), while the behavior that owns it still waits 45 / 70 sec over Tor. Every range slower than 30 sec is downloaded from two nodes and one copy is discarded.

The fix in: #15233: hand out each compact filter range once.

4. The 70 sec timeout per filter range is blind to progress

A node has to deliver 8.4 MB in 70 sec or it is disconnected and the partial download is lost. That is 120 KB/s per circuit, and with 5 ranges in flight the client needs about 600 KB/s in total or it keeps timing out and starting over. (This is what may have caused this: #14915).

The fix in #15235: reset the timer every time a filter arrives, so a slow node that is making progress is kept.

5. Node rotation evicts working nodes while 3 of 12 slots are used

ReevaluateConnectionsAsync rotates a node every 4 sec whenever fewer than 5 nodes serving compact filters are connected, regardless of how many nodes are connected in total. In run A there were 11 replacements, the first one with 3 nodes connected, and it dropped the node that was serving the headers for a candidate that closed the connection 2.4 sec later. (Related: #14770.)

Fix: rotate only when all 12 slots are full.

---

More issues will be added, as they are found and studied.

[lontivero]

This is great. Thank you. I will be able to review them next week.