legacy hyper-util client: option to not open a new connection when an idle one is about to be free

[aryehklein-rise]

When a request doesn't find a free pooled connection immediately, the legacy
Client starts opening a new connection right away and races it against
waiting for a pooled one to come back (connection_for in
src/client/legacy/client.rs). If a pooled connection comes back first, the
new connection was opened for nothing. I couldn't find a way to turn this off.

I might be misunderstanding how the pool works, so please correct me if so.

But when turning it off by changing the code my tail latency and error rates went down significantly.

[seanmonstar]

How would we know that an idle one is about to be free? What if it's not going to be free for quite some time, or ever? The connection_for future is wrapped in a lazy construct, so before it starts making a new TCP connection, it will check if an idle connection is available. That cuts down on socket churn.