ingest: implement connection balancing based on write load

[peterbourgon]

Ingesters can gossip some concept of load, e.g. connected clients, total ingest throughput, etc. With that knowledge shared, they can do some kind of connection balancing, in order of increasing severity

1. Extend the forwarder/ingester protocol to allow the ingesters to suggest other ingesters to newly connected forwarders. I think this can be as simple as having the heaviest-loaded ingesters write the address of the lightest-loaded ingesters to each new connection, and relying on the forwarder to read that data and choose to drop the connection to the one forwarder, and connect to the other one.
2. Have highly-loaded ingester/s turn off their connection listener when certain conditions are met. Those conditions must be very conservative. There must be very few ingesters in this state in a given cluster, perhaps no more than one. They must not stay that way permanently. Basically, it's always better to accept new connections to an overloaded ingester than to accidentally DoS ourselves by shutting down too many listeners.
3. Have highly-loaded ingester/s actively terminate forwarder connections. This must only occur after the connection listener is shut down, and must be even more conservative. They should probably bias to killing the most recent connections, or the ones that are doing the lowest throughput, either instantaneous or lifetime. It should not terminate connections down to zero. It should practice some form of hysteresis. And so on.