circus cluster

[tarekziade]

this is from a discussion with @kylef

what about adding a circus-cluster script that's able to communicate with several circusd instances

The circusd instance would dynamically register themselves as a new node to the circus cluster, using the heartbeat pattern as implemented in powerhose workers.

the circus-cluster would provide all commands circusctl provides, but with an extra parameter: the node id.

The circus-cluster would also allow getting the stats of each node.

[ericholscher]

👍 Thought this was something that circus did when I heard about it. It's a really neat idea.

[tarekziade]

In details:

• a circusd-cluster script, that opens a backend heartbeat socket for circusd processes to connect, and a frontend socket that's exactly like the one circusd opens
• a new option in circusctl for all commands, node id. When circusctl is connected to a socket, it recognizes if it's a node or a cluster, and accept the node option if needed
• a new command called "nodelist" that returns a list of all nodes.
• a new option in [circus]: cluster_endpoint, that let each node know where is the master
• a circusd-cluster-stats, that reunites all circusd-stats streams

[tarekziade]

I have started a branch for this feature

[almet]

It could be really helpful if we had a way to switch from handling processes for a single node to processes of a cluster, just by using a different zmq endpoint.

We could have two modes:

• A "normal" mode, which acts exactly like a node, except that circus-cluster is handling the load balancing between the nodes he manages
• An advanced one which allows the user to run circusd commands on a specific node.

The only difference would be a argument more for the "advanced" mode.

I like the flexibility the normal mode provides us: we can specify different limits (cpu-wise, memory-wise, etc) which, when met, would trigger different behaviors (go to poke another server, stop accepting new processes, etc).

[Natim]

We will first implement multicast discovery so that circushttpd can have a list of endpoints and that we can monitor all vm with the same circushttpd. It will be the first step of tomorow sprints.