Better support for layer-4 (TCP) proxies

[jhump]

When a set of servers is behind a layer-4 proxy, the default behavior of an httplb.Client will not result in meaningful load distribution. In such a case, DNS resolution typically returns a single address, a VIP for the proxy. But since it's a layer-4 proxy, it will not distribute requests to multiple backends: instead, a single connection is effectively pinned to a single backend behind the proxy.

The first thing needed to work in this scenario is a way to make the client establish multiple connections, even if to the same backend address. The layer-4 load balancer will then distribute these connections to multiple backends. That way, picking from these multiple connections (such as the default round robin algorithm) results in distribution of requests to multiple backends.

The above is not by itself sufficient because it is possible that multiple connections still have low backend diversity. This could easily happen when the total number of connections in the client is low and so is the number of backends. In this case, just by chance, the proxy could route multiple (even all) connections to the same backend instance. This can also happen in some operational scenarios, such as rolling restarts, where one node is restarted and the underlying "virtual" connection reconnects. If some of the backends are unavailable (due to the restart), there is a greater chance that these reconnects find their way to the same backend.

So, to work around the potential issue of low backend diversity, we also need a way to recycle connections. Periodically forcing "virtual connections" to re-connect. While we can't prevent the situation of low backend diversity, since there is no way to influence how a connection through a layer-4 proxy gets routed, periodically recycling connections allows the client to "heal" its connection pool if/when it does get into this state.

Re-connecting needs to be graceful, so that any in-progress operations on the old connection are allowed to complete, uninterrupted, but the new connection is used for all new operations.

[jhump]

A key use case for this is Kubernetes services. Standard services use a single virtual IP for the service, which is proxied to backend pod IPs at layer-4 (using iptables rules, instead of an actual proxy server).

[jhump]

cc @DMarby