Manual embedded etcd clustering

[portertech]

• create basic cluster (not manage)

CLI/ENV argument/variable for peers.

[grepory]

To introduce clustering to sensu-backend, we'll be using static etcd clustering.

Static clustering requires that users assign unique IDs to each member of the cluster and that each member of the cluster know about the rest of the cluster prior to bootstrapping the cluster. Each sensu-backend process must be started with the --cluster-members argument (or by setting the SENSU_CLUSTER_MEMBERS environment variable) with the CSV list of cluster members of the format member_id=scheme://host:port. This is easier if illustrated.

Given a cluster:

• a=http://a:2380
• b=http://b:2380
• c=http://c:2380

The sensu-backend processes on each machine would be started thus:

Machine a:
sensu-backend --id a --etcd-peer-url http://a:2380 --cluster-members a=http://a:2380,b=http://b:2380,c=http://c:2380

Machine b:
sensu-backend --id b --etcd-peer-url http://b:2380 --cluster-members a=http://a:2380,b=http://b:2380,c=http://c:2380

Machine c:
sensu-backend --id c --etcd-peer-url http://c:2380 --cluster-members a=http://a:2380,b=http://b:2380,c=http://c:2380

The cluster member ID of a node cannot change over the course of its life.

If a user is clustering backends, they must specify all three of these flags on the command line, because we cannot guarantee that we generate an ID that matches a peer in the provided cluster member list.

[grepory]

It's probably worth documenting this for users on Kubernetes.

https://github.com/coreos/etcd-operator/blob/master/doc/best_practices.md