re-use reqwest::Client in disk_manual_import #5750
Conversation
|
Here's a look at things uploading 64 disks in parallel from dogfood -> london In aggregate I'm seeing a pretty stable 1.1 to 1.2gbit/s on my dogfood VM
On london, nexus is only using 5-6% CPU. previously it was using 100% of a sled's CPU (for a fraction of the upload throughput)
Interesting to note that since we are processing more requests now, cockroach CPU usage is a bit higher on one of the other sleds. I think this is probably auth handling? Doesn't seem to be a problem at the moment.
Basically we are uploading like 10x faster while using 1/20th the CPU, and probably not bottlenecking in this part of the process anymore. You love a |
| )); | ||
| let client = crucible_pantry_client::Client::new_with_client( | ||
| &format!("http://{}", endpoint), | ||
| self.reqwest_client.clone(), |
There was a problem hiding this comment.
Ahh sorry, I definitely misunderstood this fix when I suggested it sounded like we needed connection pooling. Connection pooling presumably would also do something like this, but seems largely orthogonal to "don't create a million reqwest::Clients", as you pointed out!
Fixes #5717. @faithanalog will follow-up with performance numbers she's running now.
We should probably do this to other clients we generate on-the-fly in Nexus, too, but this is probably the worst offender to fix.