proposal: net/http: Allow configurable max connection age

[dmarkhas]

Currently, the http.Transport implementation will reuse an HTTP connection if keepalives are enabled and the connection has not been in an idle state for too long.
This means that for high throughput connections that are always active and rarely/never exceed the IdleConnTimeout limit, the connection will keep getting reused (as long as it is not broken).

This poses a problem in terms of service discovery and client-side load balancing - since address resolution only happens in the dialer's DialContext, this leaves no chance for the client to re-resolve the target address since the same connection is constantly reused without ever "re-dialing".

I propose to add a MaxConnectionAge flag to http.Transport that would limit the lifetime of a persistent connection, and would not reuse it beyond this lifetime.

This would be in line with the behavior of go-grpc which also enforces a MaxConnectionAge limit.

[networkimprov]

cc @bradfitz @neild

[RobGraham]

Similar to #23427 which offers a work around (hack). However re-resolving DNS and reusing connection issues have been apparent for many years.

[szuecs]

Maybe the TTL based DNS resolution could be a solution #23427 (comment).
#23427 (comment) summarized 3 possible solutions.

I am pretty sure that in practice you never max out the number of connections, so you always have idle conns that will be closed in a "hack" as I showed in #23427. We run this in a high traffic scenario since some years without an issue that we could observe (does not mean that it does not exist in practice).

[raidancampbell]

I have a different issue that would be resolved by the same configuration. For reasons outside my control, I have a firewall that kills TCP connections after 15 minutes regardless of their activity. As mentioned above MaxConnectionAge solves this server-side in gRPC.

Having a similar configuration would be an easy client-side fix for me in HTTP, but I'll readily admit this problem is not likely common enough to warrant a change to net/http.

[jpcope]

Note, I believe I have a working implementation proposal at #46714 and it may be reusable for addressing #23427

[jpcope]

Also, sorry for the potentially dupe proposal. I believe the maintainers were not interested in reviewing my PR without a proposal of my own for my specific implementation.

#54429

[akrovv]

Any updates?