Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Rework event distribution #6173
A common issue we have with clustering is the network chatter caused by our current broadcast approach to event distribution where every single member connects to all other members.
To improve this, we should switch to using event hubs, Effectively select LXD servers (ideally not a database node) which will receive events from all members and then forward those onto all other members.
The setup should allow for multiple hubs to operate at the same time with LXD servers picking one at random and hubs forwarding events between themselves.
On the sending side, an event should be blocked until it's successfully made it to a hub.
This is all hidden away from the user, the user will keep connecting to /1.0/events and get the usual websocket stream of events from any of the API servers, they do not need to directly talk to a hub as with this model, all LXD servers still receive all events, they just do so using a single websocket connection to a hub rather than having to deal with one connection per server.
This will build on top of #6172 using the
Further down the line we may look into having /1.0/events be redirected to a event hub and stop having all LXD servers receive all events or switch to a more complex subscription mechanism where LXD servers subscribe to only those events they care about, but that's work for later and the reliability/load improvements coming from the first pass on this should be good enough for the cluster sizes we're looking at now (50 nodes up to 100 nodes).
While this is a dream, will you be including the server the event came from in the response ?
If I understand correctly this is the new process:
I may have a problem that I still need to know if a "container was started" (any event really) on host
@turtle0x1 that's what the