Improve syncer

[ferranbt]

Current

Syncer runs N workers and each one queries a job (headers, bodies or receipts) to a given peer at a time. The peer selection is done by sorting all the active peers to which we are connected and sorting by number of outstanding queries (4 is the max number of outstanding queries).

Syncer keeps a queue (linked list) of query blocks. Each item in the queue is a group of 100 blocks. The query jobs are done per item, this is, the max number of elements in the headers query will be 100 (when the max number of elements we can retrieve from geth is 190). In the case of bodies and receipts, if only six blocks in the item have bodies, the bodies query will ask for only six elements. This approach underuses the network.

The only query limit factor for a given peer is the max number of outstanding queries (4). It does not take into account the congestion (time to receive the response).

Proposal

The sync scheduler creates a peer connection service for each peer. That connection will request bandwidth from the scheduler to make the queries. Then, it will ask the scheduler picker for a job. Peer connection will try to do as many as concurrent requests it can do (if there is available bandwidth). Periodically, it measures the rtt time from the queries to change the number of concurrent requests it can make. The higher the rtt the lower the number of requests.

It will periodically check the rate of each connection. If there are too many active connections competing for the bandwidth it will idle the connections with the lower rate. At the same time, if there is unused bandwidth it will active idle peers if there are any.