You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 8, 2018. It is now read-only.
Ingesters can gossip some concept of load, e.g. connected clients, total ingest throughput, etc. With that knowledge shared, they can do some kind of connection balancing, in order of increasing severity
Extend the forwarder/ingester protocol to allow the ingesters to suggest other ingesters to newly connected forwarders. I think this can be as simple as having the heaviest-loaded ingesters write the address of the lightest-loaded ingesters to each new connection, and relying on the forwarder to read that data and choose to drop the connection to the one forwarder, and connect to the other one.
Have highly-loaded ingester/s turn off their connection listener when certain conditions are met. Those conditions must be very conservative. There must be very few ingesters in this state in a given cluster, perhaps no more than one. They must not stay that way permanently. Basically, it's always better to accept new connections to an overloaded ingester than to accidentally DoS ourselves by shutting down too many listeners.
Have highly-loaded ingester/s actively terminate forwarder connections. This must only occur after the connection listener is shut down, and must be even more conservative. They should probably bias to killing the most recent connections, or the ones that are doing the lowest throughput, either instantaneous or lifetime. It should not terminate connections down to zero. It should practice some form of hysteresis. And so on.
The text was updated successfully, but these errors were encountered:
Ingesters can gossip some concept of load, e.g. connected clients, total ingest throughput, etc. With that knowledge shared, they can do some kind of connection balancing, in order of increasing severity
The text was updated successfully, but these errors were encountered: