Flaky behavior `reqwest`

[davidbtadokoro]

Description:

The project uses the crate reqwest for HTTP request, necessary for fetching patch feeds from target mailing lists. In the first step of fetching the latest patchsets from a list, a request is made to Lore API, which normally takes around 1 to 2 seconds (which in itself isn't great).

The problem is that sometimes (for reasons I can't pinpoint) the latency/speed/something on the request abnormally takes from 5 seconds to 20 seconds! Monitoring my nic, I can see that the start of the transfer hangs, as the download speed on the interface (normally) spikes to 200-1000Kb/s.

I know that the problem is on the network step, as the local processing of patches is consistently around 1 second per response. It is also strange the fact that when this happens, making the same request in the browser or with curl seems to also take a bit more than normal, but much faster than through the app.

How to reproduce:

By running the draft app with cargo run <target-list> <page-size>, you may or not experience the above (that's why it's flaky 😩).

Expected behavior:

The crate reqwest (or the wrapper code around it) shouldn't have this flakyness on performance.

Setup:

• Project branch: master
• Project commit hash: 8eac6e2
• rustc version: 1.79.0
• cargo version: 1.79.0

[davidbtadokoro]

Maybe implementing a client for subsequent request will help. I've noticed that when encountering this issue of (constantly) getting big latencies, switching IPs using a VPN provider stabilizes the request time, like "the new client from a different location). I am brainstorming here, but it can be some kind of throttling based on the client.

[auyer]

@davidbtadokoro can you mark this as solved ?

[davidbtadokoro]

Hey, @auyer. Like in kw, we wait for the stable (master branch) to close issues. So this will close as soon as we have a new release.