Peer discovery with consul and UbuntuFAN overlay network.

[frederikbosch]

In our cluster we are using UbuntuFAN as an overlay network. Given a VM with IP 172.16.50.111, all docker containers are assigned IP in the range 250.50.111.0/24, with the VM itself - the docker host - as 250.50.111.0.

I am trying to get RabbitMQ (3.13.1) working on a 3 node cluster (worker1, worker2 and worker4) that uses peer discovery via Consul.

docker:

image = "rabbitmq:3.13-management"
hostname = "worker1.rabbitmq-nomad.service.consul"

The above hostname is a FQDN that resolves to an actual IP address using Consul's DNS service.

rabbitmq.conf

log.console.level = debug
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_consul
cluster_formation.consul.svc_addr_auto = false
cluster_formation.consul.svc_addr = worker1.rabbitmq-nomad.service.consul
cluster_formation.consul.svc_tags.1 = amqp
cluster_formation.consul.svc_tags.2 = worker1
cluster_formation.consul.host = 172.16.50.111
cluster_formation.consul.acl_token = ****

enabled_plugins

[rabbitmq_management,
rabbitmq_federation,
rabbitmq_federation_management,
rabbitmq_shovel,
rabbitmq_shovel_management,
rabbitmq_mqtt,
rabbitmq_stomp,
rabbitmq_peer_discovery_consul].

env

RABBITMQ_USE_LONGNAME="true"

Now when the three workers boot, they do not detect peers. They all result in standalone instances. Those standalone instance are registering in Consul. When I enabled debug logging, I discovered that all instances are querying Consul correctly and they extract peers from the service registry.

2024-05-03 08:27:41.686079+00:00 [debug] <0.307.0> Feature flags: releasing lock {feature_flags_state_change,<0.307.0>}
2024-05-03 08:27:41.686357+00:00 [debug] <0.307.0> Feature flags: unregistering controller globally
2024-05-03 08:27:41.686479+00:00 [debug] <0.307.0> Feature flags: [global unregister] @ rabbit@worker4.rabbitmq-nomad.service.consul
2024-05-03 08:27:41.687092+00:00 [info] <0.247.0> DB: virgin node -> run peer discovery
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0> Peer discovery: translated cluster formation configuration: [{peer_discovery_backend,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                               rabbit_peer_discovery_consul},
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                              {peer_discovery_consul,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                               [{consul_host,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                 "172.16.50.131"},
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                {consul_acl_token,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                 "6f5c3ebb-8541-e8a5-fcbd-43f03bdf2976"},
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                {consul_svc_addr,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                 "worker4.rabbitmq-nomad.service.consul"},
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                {consul_svc_addr_auto,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                 false},
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                {consul_svc_tags,
2024-05-03 08:27:41.687359+00:00 [debug] <0.247.0>                                                                 ["worker4"]}]}]
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0> Peer discovery: cluster formation backend configuration: [{consul_host,
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                            "172.16.50.131"},
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                           {consul_acl_token,
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                            "6f5c3ebb-8541-e8a5-fcbd-43f03bdf2976"},
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                           {consul_svc_addr,
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                            "worker4.rabbitmq-nomad.service.consul"},
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                           {consul_svc_addr_auto,
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                            false},
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                           {consul_svc_tags,
2024-05-03 08:27:41.688137+00:00 [debug] <0.247.0>                                                            ["worker4"]}]
2024-05-03 08:27:41.715812+00:00 [debug] <0.247.0> GET http://172.16.50.131:8500/v1/health/service/rabbitmq?
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0> Response: {ok,{{"HTTP/1.1",200,"OK"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                [{"date","Fri, 03 May 2024 08:27:42 GMT"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"vary","Accept-Encoding"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"content-length","2931"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"content-type","application/json"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"x-consul-default-acl-policy","deny"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"x-consul-effective-consistency","leader"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"x-consul-index","28680408"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"x-consul-knownleader","true"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"x-consul-lastcontact","0"},
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                 {"x-consul-query-backend","blocking-query"}],
2024-05-03 08:27:42.086162+00:00 [debug] <0.247.0>                "[{\"Node\":{\"ID\":\"eacdd31f-f7c6-a9d9-f5a5-00f9b5bd0c17\",\"Node\":\"worker1\",\"Address\":\"250.50.111.1\",\"Datacenter\":\"dc-vboxx\",\"TaggedAddresses\":{\"lan\":\"250.50.111.1\",\"lan_ipv4\":\"250.50.111.1\",\"wan\":\"250.50.111.1\",\"wan_ipv4\":\"250.50.111.1\"},\"Meta\":{\"consul-network-segment\":\"\",\"consul-version\":\"1.16.6\"},\"CreateIndex\":28446895,\"ModifyIndex\":28446896},\"Service\":{\"ID\":\"rabbitmq:worker1.rabbitmq-nomad.service.consul\",\"Service\":\"rabbitmq\",\"Tags\":[\"worker1\"],\"Address\":\"worker1.rabbitmq-nomad.service.consul\",\"Meta\":null,\"Port\":5672,\"Weights\":{\"Passing\":1,\"Warning\":1},\"EnableTagOverride\":false,\"Proxy\":{\"Mode\":\"\",\"MeshGateway\":{},\"Expose\":{}},\"Connect\":{},\"PeerName\":\"\",\"CreateIndex\":28680335,\"ModifyIndex\":28680335},\"Checks\":[{\"Node\":\"worker1\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"ServiceTags\":[],\"Type\":\"\",\"Interval\":\"\",\"Timeout\":\"\",\"ExposedPort\":0,\"Definition\":{},\"CreateIndex\":28446895,\"ModifyIndex\":28446895},{\"Node\":\"worker1\",\"CheckID\":\"service:rabbitmq:worker1.rabbitmq-nomad.service.consul\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Consul-based peer discovery plugin TTL check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq:worker1.rabbitmq-nomad.service.consul\",\"ServiceName\":\"rabbitmq\",\"ServiceTags\":[\"worker1\"],\"Type\":\"ttl\",\"Interval\":\"\",\"Timeout\":\"\",\"ExposedPort\":0,\"Definition\":{},\"CreateIndex\":28680335,\"ModifyIndex\":28680335}]},{\"Node\":{\"ID\":\"1a73b3f6-e573-f5e4-6442-d054ab7e78be\",\"Node\":\"worker2\",\"Address\":\"250.50.112.1\",\"Datacenter\":\"dc-vboxx\",\"TaggedAddresses\":{\"lan\":\"250.50.112.1\",\"lan_ipv4\":\"250.50.112.1\",\"wan\":\"250.50.112.1\",\"wan_ipv4\":\"250.50.112.1\"},\"Meta\":{\"consul-network-segment\":\"\",\"consul-version\":\"1.16.6\"},\"CreateIndex\":28446925,\"ModifyIndex\":28446926},\"Service\":{\"ID\":\"rabbitmq:worker2.rabbitmq-nomad.service.consul\",\"Service\":\"rabbitmq\",\"Tags\":[\"worker2\"],\"Address\":\"worker2.rabbitmq-nomad.service.consul\",\"Meta\":null,\"Port\":5672,\"Weights\":{\"Passing\":1,\"Warning\":1},\"EnableTagOverride\":false,\"Proxy\":{\"Mode\":\"\",\"MeshGateway\":{},\"Expose\":{}},\"Connect\":{},\"PeerName\":\"\",\"CreateIndex\":28680331,\"ModifyIndex\":28680331},\"Checks\":[{\"Node\":\"worker2\",\"CheckID\":\"serfHealth\",\"Name\":\"Serf Health Status\",\"Status\":\"passing\",\"Notes\":\"\",\"Output\":\"Agent alive and reachable\",\"ServiceID\":\"\",\"ServiceName\":\"\",\"ServiceTags\":[],\"Type\":\"\",\"Interval\":\"\",\"Timeout\":\"\",\"ExposedPort\":0,\"Definition\":{},\"CreateIndex\":28446925,\"ModifyIndex\":28446925},{\"Node\":\"worker2\",\"CheckID\":\"service:rabbitmq:worker2.rabbitmq-nomad.service.consul\",\"Name\":\"Service 'rabbitmq' check\",\"Status\":\"passing\",\"Notes\":\"RabbitMQ Consul-based peer discovery plugin TTL check\",\"Output\":\"\",\"ServiceID\":\"rabbitmq:worker2.rabbitmq-nomad.service.consul\",\"ServiceName\":\"rabbitmq\",\"ServiceTags\":[\"worker2\"],\"Type\":\"ttl\",\"Interval\":\"\",\"Timeout\":\"\",\"ExposedPort\":0,\"Definition\":{},\"CreateIndex\":28680331,\"ModifyIndex\":28680341}]}]"}}
2024-05-03 08:27:42.106347+00:00 [debug] <0.247.0> Peer discovery: backend returned the following configuration:
2024-05-03 08:27:42.106347+00:00 [debug] <0.247.0>   {ok,{['rabbit@worker1.rabbitmq-nomad.service.consul',
2024-05-03 08:27:42.106347+00:00 [debug] <0.247.0>         'rabbit@worker2.rabbitmq-nomad.service.consul'],
2024-05-03 08:27:42.106347+00:00 [debug] <0.247.0>        disc}}
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0> Peer discovery: peer node arguments: #{args =>
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0>                                            ["-setcookie","28&#$d4V726%Ny",
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0>                                             "-boot","start_sasl","-hidden"],
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0>                                        name => "rabbit-4-18",
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0>                                        host =>
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0>                                            "worker4.rabbitmq-nomad.service.consul",
2024-05-03 08:27:42.106755+00:00 [debug] <0.247.0>                                        longnames => true}
2024-05-03 08:27:42.812860+00:00 [debug] <0.247.0> Peer discovery: using temporary hidden node 'rabbit-4-18@worker4.rabbitmq-nomad.service.consul' to query discovered peers properties
2024-05-03 08:27:42.868750+00:00 [debug] <36761.99.0> Peer discovery: sorted list of nodes and their properties considered to create/sync the cluster:
2024-05-03 08:27:42.868750+00:00 [debug] <36761.99.0>   - {'rabbit@worker2.rabbitmq-nomad.service.consul',['rabbit@worker2.rabbitmq-nomad.service.consul'],1714724421789318,true}
2024-05-03 08:27:42.868750+00:00 [debug] <36761.99.0>   - {'rabbit@worker1.rabbitmq-nomad.service.consul',['rabbit@worker1.rabbitmq-nomad.service.consul'],1714724424666388,true}
2024-05-03 08:27:42.871603+00:00 [debug] <0.247.0> Peer discovery: not satisfyied with discovered peers: the list does not contain this node
2024-05-03 08:27:42.871867+00:00 [debug] <0.247.0> Peer discovery: retrying to create/sync cluster in 1000 ms (29 attempts left)
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0> Peer discovery: translated cluster formation configuration: [{peer_discovery_backend,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                               rabbit_peer_discovery_consul},
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                              {peer_discovery_consul,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                               [{consul_host,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                 "172.16.50.131"},
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                {consul_acl_token,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                 "6f5c3ebb-8541-e8a5-fcbd-43f03bdf2976"},
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                {consul_svc_addr,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                 "worker4.rabbitmq-nomad.service.consul"},
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                {consul_svc_addr_auto,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                 false},
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                {consul_svc_tags,
2024-05-03 08:27:43.873305+00:00 [debug] <0.247.0>                                                                 ["worker4"]}]}]
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0> Peer discovery: cluster formation backend configuration: [{consul_host,
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                            "172.16.50.131"},
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                           {consul_acl_token,
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                            "6f5c3ebb-8541-e8a5-fcbd-43f03bdf2976"},
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                           {consul_svc_addr,
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                            "worker4.rabbitmq-nomad.service.consul"},
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                           {consul_svc_addr_auto,
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                            false},
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                           {consul_svc_tags,
2024-05-03 08:27:43.873963+00:00 [debug] <0.247.0>                                                            ["worker4"]}]

It says Peer discovery: not satisfyied with discovered peers: the list does not contain this node. But I do not see this node registering in Consul at that specific moment. The node registers in consul when it has started as a standalone node, not earlier on.

After the nodes have booted (as standalone instance), I have tried to let them join another node manually.

root@worker2:/# rabbitmq-diagnostics resolve_hostname worker1.rabbitmq.service.consul --address-family IPv4 --offline
Resolving hostname worker1.rabbitmq.service.consul to IPv4 addresses...
250.50.111.7
root@worker2:/# rabbitmqctl join_cluster rabbit@250.50.111.7
Clustering node rabbit@worker2.rabbitmq-nomad.service.consul with rabbit@250.50.111.7

08:52:41.240 [error] Feature flags: error while running:
Feature flags:   rabbit_ff_controller:running_nodes[]
Feature flags: on node `rabbit@250.50.111.7`:
Feature flags:   exception error: {erpc,noconnection}
Feature flags:     in function  erpc:call/5 (erpc.erl, line 710)
Feature flags:     in call from rabbit_ff_controller:rpc_call/5 (rabbit_ff_controller.erl, line 1378)
Feature flags:     in call from rabbit_ff_controller:list_nodes_clustered_with/1 (rabbit_ff_controller.erl, line 478)
Feature flags:     in call from rabbit_ff_controller:check_node_compatibility_task/2 (rabbit_ff_controller.erl, line 390)
Feature flags:     in call from rabbit_db_cluster:can_join/1 (rabbit_db_cluster.erl, line 59)
Feature flags:     in call from rabbit_db_cluster:join/2 (rabbit_db_cluster.erl, line 91)
Feature flags:     in call from erpc:execute_call/4 (erpc.erl, line 589)

Error:
{:aborted_feature_flags_compat_check, {:error, {:erpc, :noconnection}}}

Whether I use IP or FQDN does not matter. The result is the same. If I install telnet in the container and I try to connect to the EPMD port on the other node, this connection can be established. The other node (worker1 in this case) does not show any logs regarding the try of worker2 to join its cluster.

root@worker2:/# telnet 250.50.111.7 4369
Trying 250.50.111.7...
Connected to 250.50.111.7.
Escape character is '^]'.

So even if I would be registering the rabbitmq service in Consul beforehand, using Nomad, and as such make sure this node is in the list, then connecting and joining fails anyhow. I ran out of options, so hopefully someone here can help out.

[frederikbosch]

Okay, a little update, running the above configuration does work with

image = "rabbitmq:3.12-management"

When running on 3.12.13 I discovered that my consul acl token was lacking some privileges in the underlying policy. I did not have session write and key write on the rabbitmq/ prefix. There is no error with that when running on 3.13. After updating the policy, my configuration works on 3.12.13, no connection issues and no the list does not contain this node. The nodes form a cluster correctly.

Retrying with the correct policy on 3.13 does not give any different results. As a side note, my recommendation would be to document the required consul policy. I am willing to create a PR for this.

But my original post still stands.

[frederikbosch]

Okay, I believe I hit what will be resolved via #11045.

[dumbbell]

Thank you @frederikbosch for the detailed analysis and the ping in #11045 as I didn't see your question yet :-)

I don't know Consul well; in fact, I discover how it works while working on this regression. I will re-read what you wrote carefully and may come back with questions to better understand your use case.

[frederikbosch]

If you´d like, I could test your PR once finished.

Below a screenshot from a RabbitMQ service I registered in Consul with Nomad. My guess is that people using container orchestration like Nomad, Kubernetes or Docker Swarm would not require RabibtMQ to do the registration, but rather let RabbitMQ read the already registered services (from Consul or other service registry)

If you would proceed with that suggestion, you have to take into account that the service's IP might registered in the service registry, and not the FQDN of the RabbitMQ node. This would mean that RabbitMQ has to figure out itself which of the listed nodes is the this node by connecting to all the listed addresses. Another solution might be to require a meta-key to be populated with the FQDN. The latter is no problem if using Nomad, and my guess it should be no problem with another orchestration tool.

So as I see it.

1. Boot
2. Register service in Consul or skip if config indicates no service registration by RabbitMQ
3. Query Consul for nodes
4. Determine which node is this node (by connecting or meta-key in registry)
5. Construct cluster and elect leader

[dumbbell]

Thank you for the explanation! I can probably add a configuration parameter to skip registration, though not in the mentionned pull request as it's unrelated.

We need the node names of all nodes, not just this node. Anyway, we can't connect to Erlang nodes without knowing their actual Erlang node names. That's part of the connection handshake and basic security of Erlang distribution. We could have a special configurable behavior for Consul that uses reverse DNS lookups to determine the hostnames based on the IP addresses. However, I'm almost sure this would be a low priority. Would you like to contribute such a feature?

In the meantime, it would be the responsibility of the orchestration tool to set the erlang-node-name meta property for each RabbitMQ service. That's what the mentionned pull request sets during registration. Note that this meta property would only be used by RabbitMQ version having this pull request (so some future 3.13.x). It won't be back-ported to previous release branches.

[frederikbosch]

The erlang-node-name meta property is absolutely perfect. No need for reverse IP lookup then. Regarding skipping registration, I would prefer that yes. Do you want me to create an issue for that?

Is there a Docker image that I can test the PR with? Or should I build one myself?

[mkuratczyk]

image is available: pivotalrabbitmq/rabbitmq:peer-disc-register-before-discover

the image will change if there are new commits on this branch (meaning, this image reference always points to the most recent successful build from the branch specified in the tag)

[dumbbell]

@frederikbosch: Yes, filing an issue would be nice already, where you describe your use case! If you contribute code as well, we can QA it quickly. Otherwise, we probably won't prioritize it soon.

[kochen]

Hey @frederikbosch,
We are facing a similar issue, using consul and we are trying to avoid using static ports (and host locking) if possible.

were you eventually able to sort this out?

M.

[frederikbosch]

We are using the Ubuntu-FAN overlay network. It means that all containers in our cluster can automatically connect to each other. This happens through a sort of IP translation. When a VM has IP 172.16.20.101, the Docker containers on that machine use 250.20.101.x. We register these overlay network IPs in Consul. So when I connect to rabbitmq.service.consul, it translates to 250.20.101.1. When I connect from 250.20.102.1 (VM with IP 172.16.20.102), the overlay network already knows how to establish these connections. No service mesh required.

Since Ubuntu has recently decided to discontinue efforts for Ubuntu-FAN, we are planning to switch to something else. We are considering MicroOVN but I haven't had the time to really investigate. It might be an opportunity to also start working with CNI. I also saw some updates in Nomad there recently.

[kochen]

so eventually you are still connecting from a client with <server_ip>:5672, right?
given you have 3 members of the cluster, how do you select the designated <server_ip>?

[frederikbosch]

I don't care which one, all servers allow me to push messages to queues or fetch from the queues.

[kochen]

that's true - not what I meant though :)

(one of) the advantage(s) of consul is the internal dns service it offers for registered services, allowing you to contact (i,e) rabbitmq.service.consul and it will return on of the running instances for that rabbitmq service.
Unless you request (i,e) rabbit-1.rabbitmq.service.consul which will resolve to the specific instance.

So the question is, are your clients (those that need to push/read messages from the queue) connecting to rabbitmq.service.consul or are they still using the connect feature (as I showed above)?

[frederikbosch]

We are always using DNS, i.e. my clients connect to rabbitmq.service.consul.