Support for multiple etcd hosts

[ghost]

Issue: Clients using etcd typically know about multiple etcd hosts for HA reasons. There does not appear to be a way to specify multiple etcd hosts in this plugin (unlike the autocluster plugin).

Workaround: This has lead me to put etcd behind haproxy for rabbitmq, which I would prefer to avoid.

Solution: Implement support for multiple etcd hosts in the rabbitmq config.

[michaelklishin]

What would that look like? I don't see any support for multiple hosts in the autocluster version, perhaps autocluster_httpc does something less-than-obvious but the version in peer-discovery-common should be no different.

[ghost]

https://github.com/rabbitmq/rabbitmq-autocluster/blob/v3.6.x/src/autocluster_etcd.erl#L31-L42

A proper client will accept a list of servers to try to connect to and once connected it will ask for a list of nodes in the cluster. It maintains that list and tries different servers when the current connection fails.

1. Input list of initial servers from client
2. Connect to the first addess in the list (if not active, try next connection)
3. Ask cluster for list of connected servers and maintain that list actively in memory
4. If client/server connection breaks, goto step 2

[michaelklishin]

@SamYaple I'm not sure if the linked code fragment does what you claim it does. It retrieves the list of RabbitMQ nodes, not etcd nodes. There are no persistent connections to etcd in the HTTP1.1-based v2 either, so I suspect item 4 means "if an HTTP request fails, try the next host".

Supporting multiple hosts in the config file should be reasonably straightforward. However, we would like to do it in a way that would be easy to extend to similar HTTP-based discovery plugins, namely Kubernetes and Consul.

[michaelklishin]

Depends on #22 and the transition to the v3 protocol (API).

[michaelklishin]

The WIP branch of #22 already supports multiple etcd endpoints:

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_etcd

cluster_formation.etcd.endpoints.1 = localhost:2379
cluster_formation.etcd.endpoints.2 = localhost:2389
cluster_formation.etcd.endpoints.3 = localhost:2399

Previously used hostname and port settings will be converted to an additional endpoint, if provided:

cluster_formation.etcd.host = localhost
cluster_formation.etcd.port = 2379

cluster_formation.etcd.endpoints.2 = localhost:2389
cluster_formation.etcd.endpoints.3 = localhost:2399

[michaelklishin]

Closing since it's been addressed early in the implementation of #22 (see #25).