cluster mode prompt"The error was: \"hostvars['rabbitmq-master']\" is undefined"

[charnet1019]

ansible: 2.12.6

error info:

TASK [rabbitmq : [RabbitMQ] Define cluster nodes] **************************************************************************************************************************************************************
fatal: [192.168.60.8 -> localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: \"hostvars['rabbitmq-master']\" is undefined\n\nThe error appears to be in '/home/ansible/roles/rabbitmq/tasks/config_server.yml': line 30, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"[RabbitMQ] Define cluster nodes\"\n  ^ here\n"}

inventory hosts:

[demo]
#192.168.60.9 ansible_ssh_user="root" ansible_ssh_pass="secret" ansible_ssh_port=5022
192.168.60.9 hostname="test-server"

[rabbitmq_master]
192.168.60.9 hostname="rabbitmq-master"

[rabbitmq_slave]
192.168.60.8 hostname="mysql"
#192.168.60.11 hostname="demo6011"

playbook:

---

- hosts: rabbitmq_master
  gather_facts: yes

  roles:
    - role: erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rabbitmq
      vars:
        # 设置为主节点
        rabbitmq_is_master: true
        #rabbitmq_series: 3.8
        rabbitmq_series: 3.9
        #rabbitmq_series_rpm_version: 3.8.30
        #rabbitmq_series_rpm_version: 3.9.10
        rabbitmq_series_rpm_version: 3.9.16
        #AMQP端口
        rabbitmq_erlang_config: |
          {rabbit, [
             {tcp_listeners, [{"{{ ansible_default_ipv4['address'] }}", 28861}]}
            ]
          }
        rabbitmq_env_config:
          NODENAME: "{{ ansible_nodename }}"
          RABBITMQ_MNESIA_BASE: "/data/rabbitmq/data"    # 定义MQ数据目录
          RABBITMQ_LOG_BASE: "/data/rabbitmq/logs"       # 定义MQ日志目录
          #USE_LONGNAME: "true"
        rabbitmq_node_name: "{{ ansible_nodename }}"
        rabbitmq_users_to_create:
          - user: admin
            password: admin
            vhost: /
            configure_priv: .*
            read_priv: .*
            write_priv: .*
            tags: administrator
        rabbitmq_users_to_delete:
          - guest
        rabbitmq_plugins_to_enable:
          - rabbitmq_management
          - rabbitmq_shovel
        rabbitmq_management_port: 15672    # web管理端口
        rabbitmq_hide_log: false
  tags:
    - rabbitmq-master



- hosts: rabbitmq_slave
  gather_facts: yes

  roles:
    - role: erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rabbitmq
      vars:
        # 设置为从节点
        rabbitmq_slave_of: rabbitmq-master
        #rabbitmq_series: 3.8
        rabbitmq_series: 3.9
        #rabbitmq_series_rpm_version: 3.8.30
        #rabbitmq_series_rpm_version: 3.9.10
        rabbitmq_series_rpm_version: 3.9.16
        #AMQP端口
        rabbitmq_erlang_config: |
          {rabbit, [
             {tcp_listeners, [{"{{ ansible_default_ipv4['address'] }}", 28861}]}
            ]
          }
        rabbitmq_env_config:
          NODENAME: "{{ ansible_nodename }}"
          RABBITMQ_MNESIA_BASE: "/data/rabbitmq/data"    # 定义MQ数据目录
          RABBITMQ_LOG_BASE: "/data/rabbitmq/logs"       # 定义MQ日志目录
          #USE_LONGNAME: "true"
        rabbitmq_node_name: "{{ ansible_nodename }}"
        rabbitmq_users_to_create:
          - user: admin
            password: admin
            vhost: /
            configure_priv: .*
            read_priv: .*
            write_priv: .*
            tags: administrator
        rabbitmq_users_to_delete:
          - guest
        rabbitmq_plugins_to_enable:
          - rabbitmq_management
          - rabbitmq_shovel
        rabbitmq_management_port: 15672    # web管理端口
        rabbitmq_hide_log: false
  tags:
    - rabbitmq-slave

[rockandska]

rabbitmq_master in inventory vs rabbitmq-master in rabbitmq_slave_of

[charnet1019]

The same configuration didn't work either.
Error:

TASK [rabbitmq : [RabbitMQ] Define cluster nodes] *****************************************************************************************************************************************************************
task path: /home/ansible/roles/rabbitmq/tasks/config_server.yml:30
fatal: [192.168.60.8 -> localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: \"hostvars['rabbitmq_master']\" is undefined\n\nThe error appears to be in '/home/ansible/roles/rabbitmq/tasks/config_server.yml': line 30, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: \"[RabbitMQ] Define cluster nodes\"\n  ^ here\n"}

inventory:

[demo]
#192.168.60.9 ansible_ssh_user="root" ansible_ssh_pass="secret" ansible_ssh_port=5022
192.168.60.9 hostname="test-server"

[rabbitmq_master]
192.168.60.9 hostname="rabbitmq_master"

[rabbitmq_slave]
192.168.60.8 hostname="mysql"
#192.168.60.11 hostname="demo6011"

playbook:

---

- hosts: rabbitmq_master
  gather_facts: yes

  roles:
    - role: erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rabbitmq
      vars:
        # 设置为主节点
        rabbitmq_is_master: true
        #rabbitmq_series: 3.8
        rabbitmq_series: 3.9
        #rabbitmq_series_rpm_version: 3.8.30
        #rabbitmq_series_rpm_version: 3.9.10
        rabbitmq_series_rpm_version: 3.9.16
        #AMQP端口
        rabbitmq_erlang_config: |
          {rabbit, [
             {tcp_listeners, [{"{{ ansible_default_ipv4['address'] }}", 28861}]}
            ]
          }
        rabbitmq_env_config:
          NODENAME: "{{ ansible_nodename }}"
          RABBITMQ_MNESIA_BASE: "/data/rabbitmq/data"    # 定义MQ数据目录
          RABBITMQ_LOG_BASE: "/data/rabbitmq/logs"       # 定义MQ日志目录
          #USE_LONGNAME: "true"
        rabbitmq_node_name: "{{ ansible_nodename }}"
        rabbitmq_users_to_create:
          - user: admin
            password: admin
            vhost: /
            configure_priv: .*
            read_priv: .*
            write_priv: .*
            tags: administrator
        rabbitmq_users_to_delete:
          - guest
        rabbitmq_plugins_to_enable:
          - rabbitmq_management
          - rabbitmq_shovel
        rabbitmq_management_port: 15672    # web管理端口
        rabbitmq_hide_log: false
  tags:
    - rabbitmq-master



- hosts: rabbitmq_slave
  gather_facts: yes

  roles:
    - role: erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rabbitmq
      vars:
        # 设置为从节点
        rabbitmq_slave_of: rabbitmq_master
        #rabbitmq_series: 3.8
        rabbitmq_series: 3.9
        #rabbitmq_series_rpm_version: 3.8.30
        #rabbitmq_series_rpm_version: 3.9.10
        rabbitmq_series_rpm_version: 3.9.16
        #AMQP端口
        rabbitmq_erlang_config: |
          {rabbit, [
             {tcp_listeners, [{"{{ ansible_default_ipv4['address'] }}", 28861}]}
            ]
          }
        rabbitmq_env_config:
          NODENAME: "{{ ansible_nodename }}"
          RABBITMQ_MNESIA_BASE: "/data/rabbitmq/data"    # 定义MQ数据目录
          RABBITMQ_LOG_BASE: "/data/rabbitmq/logs"       # 定义MQ日志目录
          #USE_LONGNAME: "true"
        rabbitmq_node_name: "{{ ansible_nodename }}"
        rabbitmq_users_to_create:
          - user: admin
            password: admin
            vhost: /
            configure_priv: .*
            read_priv: .*
            write_priv: .*
            tags: administrator
        rabbitmq_users_to_delete:
          - guest
        rabbitmq_plugins_to_enable:
          - rabbitmq_management
          - rabbitmq_shovel
        rabbitmq_management_port: 15672    # web管理端口
        rabbitmq_hide_log: false
  tags:
    - rabbitmq-slave

[rockandska]

Answered too fast.

rabbitmq_master rabbitmq_slave are groups.
rabbitmq_slave_of need to be a reference to an inventory_hostname (192.168.60.8 in your example)

[charnet1019]

It's not that I'm too quick to answer, it's that I've spent 3 days on this question and and tried other ways.
Now no error appear and MQ service running successfully when special rabbitmq_slave_of to mysql in playbook, but the cluster was not formed successfully.

Does rabbitmq_slave_of point to the master hostname or the slave hostname. Or whether other parameters need to be specified?
Thanks.

Inventory:

[rabbitmq_master]
rabbitmq-master ansible_host=192.168.60.9

[rabbitmq_slave]
mysql ansible_host=192.168.60.8
#192.168.60.11 hostname="demo6011"

[root@rabbitmq-master rabbitmq]# rabbitmqctl cluster_status
Cluster status of node rabbitmq-master@rabbitmq-master ...
Basics

Cluster name: rabbitmq-master@rabbitmq-master

Disk Nodes

rabbitmq-master@rabbitmq-master

Running Nodes

rabbitmq-master@rabbitmq-master

Versions

rabbitmq-master@rabbitmq-master: RabbitMQ 3.8.30 on Erlang 23.3.4.11

Maintenance status

Node: rabbitmq-master@rabbitmq-master, status: not under maintenance

Alarms

(none)

Network Partitions

(none)

Listeners

Node: rabbitmq-master@rabbitmq-master, interface: [::], port: 15672, protocol: http, purpose: HTTP API
Node: rabbitmq-master@rabbitmq-master, interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Node: rabbitmq-master@rabbitmq-master, interface: 192.168.60.9, port: 28861, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Feature flags

Flag: drop_unroutable_metric, state: enabled
Flag: empty_basic_get_metric, state: enabled
Flag: implicit_default_bindings, state: enabled
Flag: maintenance_mode_status, state: enabled
Flag: quorum_queue, state: enabled
Flag: user_limits, state: enabled
Flag: virtual_host_metadata, state: enabled

[rockandska]

Do the hosts specified in the rabbitmq configuration could be resolved ?
Does rabbitmq-master could resolve mysql ?
Does mysql could resolve rabbitmq-master ?

[charnet1019]

Yes, can resolved.

In the playbook I did not find the operation of the node joining the cluster.
Whether to call the command line to perform the cluster join operation in the playbook?
Thanks.

mysql host:

[root@mysql rabbitmq]# hostname
mysql
[root@mysql rabbitmq]# ping mysql
PING mysql (192.168.60.8) 56(84) bytes of data.
64 bytes from mysql (192.168.60.8): icmp_seq=1 ttl=64 time=0.029 ms
64 bytes from mysql (192.168.60.8): icmp_seq=2 ttl=64 time=0.022 ms
64 bytes from mysql (192.168.60.8): icmp_seq=3 ttl=64 time=0.023 ms
^C64 bytes from mysql (192.168.60.8): icmp_seq=4 ttl=64 time=0.027 ms
^C
--- mysql ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.022/0.025/0.029/0.004 ms
[root@mysql rabbitmq]# 
[root@mysql rabbitmq]# ping rabbitmq-master
PING rabbitmq-master (192.168.60.9) 56(84) bytes of data.
64 bytes from test-server (192.168.60.9): icmp_seq=1 ttl=64 time=0.243 ms
64 bytes from test-server (192.168.60.9): icmp_seq=2 ttl=64 time=0.346 ms
^C
--- rabbitmq-master ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.243/0.294/0.346/0.054 ms

rabbitmq-master host:

[root@rabbitmq-master ~]# hostname
rabbitmq-master
[root@rabbitmq-master ~]# ping mysql
PING mysql (192.168.60.8) 56(84) bytes of data.
64 bytes from mysql (192.168.60.8): icmp_seq=1 ttl=64 time=0.295 ms
64 bytes from mysql (192.168.60.8): icmp_seq=2 ttl=64 time=0.314 ms
64 bytes from mysql (192.168.60.8): icmp_seq=3 ttl=64 time=0.694 ms
^M64 bytes from mysql (192.168.60.8): icmp_seq=4 ttl=64 time=0.272 ms
^C
--- mysql ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3104ms
rtt min/avg/max/mdev = 0.272/0.393/0.694/0.175 ms
[root@rabbitmq-master ~]# 
[root@rabbitmq-master ~]# ping rabbitmq-master
PING rabbitmq-master (192.168.60.9) 56(84) bytes of data.
64 bytes from test-server (192.168.60.9): icmp_seq=1 ttl=64 time=0.032 ms
^C
--- rabbitmq-master ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.032/0.032/0.032/0.000 ms

[rockandska]

Without a reproducible case, without logs, hard to say what the matter is.
Try with a minimal configuration to see if it is not related to our configuration or at least provide me a full playbook who create the instances to be able to reproduce your use case and see what's going on

Regards,

[charnet1019]

Hi, I tried a few more times and it still doesn't work.

Minimal configuration
inventory hosts:

rabbitmq-master ansible_host=192.168.60.9
mysql ansible_host=192.168.60.8

[rabbitmq_master]
rabbitmq-master

[rabbitmq_slave]
mysql

playbook

---

- hosts: rabbitmq_master
  gather_facts: yes

  roles:
    - role: rockandska.erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rockandska.rabbitmq
      vars:
        rabbitmq_is_master: true
        rabbitmq_series: 3.8
        rabbitmq_series_rpm_version: 3.8.30
  tags:
    - rabbitmq-master



- hosts: rabbitmq_slave
  gather_facts: yes

  roles:
    - role: rockandska.erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rockandska.rabbitmq
      vars:
        rabbitmq_slave_of: mysql
        rabbitmq_series: 3.8
        rabbitmq_series_rpm_version: 3.8.30
  tags:
    - rabbitmq-slave

/etc/hosts

192.168.60.8 mysql
192.168.60.9 rabbitmq-master

logs
rabbitmq-playbook-output.txt
rabbitmq-service-log.txt

[charnet1019]

Resolved.
Thanks

inventory

[rabbitmq_master]
rabbitmq-master ansible_host=192.168.60.9

[rabbitmq_slave]
rabbitmq-slave ansible_host=192.168.60.8

playbook

---

- hosts: rabbitmq_master
  gather_facts: yes

  roles:
    - role: rockandska.erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rockandska.rabbitmq
      vars:
        rabbitmq_is_master: true
        rabbitmq_series: 3.8
        rabbitmq_series_rpm_version: 3.8.30
  tags:
    - rabbitmq-master



- hosts: rabbitmq_slave
  gather_facts: yes

  roles:
    - role: rockandska.erlang
      vars:
        erlang_series: 23
        erlang_series_rpm_version: 23.3.4.11-1.el7

    - role: rockandska.rabbitmq
      vars:
        rabbitmq_slave_of: rabbitmq-master
        rabbitmq_series: 3.8
        rabbitmq_series_rpm_version: 3.8.30
  tags:
    - rabbitmq-slave

And add a task config_cluster.yml

---
# Add the slave node to master cluster

- name: Stop rabbitmq app
  shell: rabbitmqctl stop_app
  when:
    - rabbitmq_slave_of is not none
    - rabbitmq_slave_of | trim | length > 0

- name: "[RabbitMQ] Define master cluster node"
  delegate_to: localhost
  set_fact:
    __rabbitmq_cluster_nodes: |
      {{
        ( __rabbitmq_cluster_nodes
          | default([],True)
        )
        + [
            node if '@' in node else ([ node, hostvars[item]['ansible_hostname'] ] | join('@'))
        ]
      }}
  loop:
    "{{
      (
        (
          groups['all']
            | map('extract',hostvars)
            | selectattr('rabbitmq_slave_of','defined')
            | selectattr('rabbitmq_slave_of','equalto',(rabbitmq_slave_of | d(inventory_hostname, true)))
            | map(attribute='inventory_hostname')
            | list
        )
        +
        [ rabbitmq_slave_of | ternary(rabbitmq_slave_of,inventory_hostname) ]
      ) | unique
    }}"
  loop_control:
    index_var: id
  vars:
    node: "{{ hostvars[item]['rabbitmq_node_name'] | d('rabbit',True) }}"
  when:
    - rabbitmq_is_master

- name: Add the slave node to cluster
  shell: rabbitmqctl join_cluster {{ __rabbitmq_cluster_nodes[0] }}
  when:
    - rabbitmq_slave_of is not none
    - rabbitmq_slave_of | trim | length > 0
  register: join_cluster_output

- name: Skip fail if the node is already a member of the cluster
  fail: msg="join_cluster failed but node is not already a member"
  when:
    - rabbitmq_slave_of is not none
    - rabbitmq_slave_of | trim | length > 0
    - ("'already_member' not in join_cluster_output.stderr") and
        (join_cluster_output.rc != 0)

- name: Start rabbitmq app
  shell: rabbitmqctl start_app
  when:
    - rabbitmq_slave_of is not none
    - rabbitmq_slave_of | trim | length > 0

[rockandska]

There is no need for extra tasks for the cluster to being functional as the nodes join the cluster automatically on first start as describded in the rabbitmq documentation.
Additionnaly there is a molecule scenario who test the cluster formation of this role and was confirmed as functional by other users

[charnet1019]

Yes, you are right.
I found out that it takes effect when rabbitmq_sysctl_config has to be configured, but the documentation doesn't say this.

[rockandska]

I found out that it takes effect when rabbitmq_sysctl_config has to be configured, but the documentation doesn't say this.

There's indeed a "bug" here because the config is not copied if no sysctl config exists but cluster mode is activated and need to fix this. (#31 created for this)

Thanks for pointing it out