Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Remove recovery threadpools and throttle outgoing recoveries on the master #15372
Today we throttle recoveries only for incoming recoveries. Nodes that have a lot
The right solution here is to also throttle the outgoing recoveries that are today unbounded on
The concurrency aspects of the recovery source also added a lot of complexity and additional threadpools that are hard to configure. This commit removes the concurrent streams notion completely and sends files in the thread that drives the recovery simplifying the recovery code considerably.
Note: this PR is still a bit rough but all tests pass and it contains a lot of improvements. I just wanted to get a pair of eyes on it for initial feedback.
I really like how this cleans up the RecoverySourceHandler!
One thing I don't understand is how many times the
Yeah so previously we had one thread driving the recovery on the source that passed all file sending task to the threadpools and then blocked until all files where transferred. Now we don't have the threadpools and just use that driver thread to send all the files sequentially. We only use throttling to limit the rate of how fast we are sending. Now with the changes on the allocation end we also limit how many senders we have per node to not overload the node. But essentially we use a single thread for this IO bound problem instead of multiple.
thats a nice side-effect of this simplification