OpenStack High Availability Dev RabbitMQ Clustering

rochfordk edited this page Jul 25, 2013 · 1 revision
Clone this wiki locally

This document outlines the steps required to spin up a multi-node clustered RabbitMQ installation using the Chef cookbook, VirtualBox and Vagrant.

Dependencies

  • git
  • VirtualBox (available from Downloads or apt-get install virtualbox)
  • Vagrant 1.2 or later (available from Downloads)
  • A Vagrant base box that includes the Chef Client e.g. opscode-ubuntu-12.04 (* 418MB download *)

Environment

Create a project folder

$ mkdir RabbitCluster

$ cd RabbitCluster/

$ vagrant init

Note that we must be able to resolve the names of the RabbitMQ servers for clustering to work. We're using the Vagrant host-manager plugin to ensure that this is the case

$ vagrant plugin install vagrant-hostmanager

Use wget or 'curl -O' to replace the generated Vagrant file

$ mv Vagrantfile Vagrantfile.orig

$ wget https://raw.github.com/rochfordk/OpenStack-HA/master/RabbitMQ/test/Vagrantfile

Install the Vagrant base box

$ vagrant box add ubuntu1204-chef path-or-url-to.box

Download the RabbitMQ cookbook (and it's dependencies).

N.B. these commands clone a modified RabbitMQ cookbook from this repository but clone the dependencies from the Opscode community repo.

$ mkdir cookbooks

$ cd cookbooks/

$ git clone https://github.com/rochfordk/rabbitmq.git
$ git clone https://github.com/opscode-cookbooks/erlang.git
$ git clone https://github.com/opscode-cookbooks/apt.git
$ git clone https://github.com/opscode-cookbooks/yum.git
$ git clone https://github.com/opscode-cookbooks/build-essential.git

$ cd ..

Configuration

VirtualBox networking

This test configuration uses VirtualBox host-only networking.

Create a host-only interface:

$ vboxmanage hostonlyif create
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interface 'vboxnet0' was successfully created

Configure the subnet using the same subnet used in out Vagrant file:

$ VBoxManage hostonlyif ipconfig vboxnet0 --ip 10.125.0.1 --netmask 255.255.255.0

And make sure there is no DHCP server running for vboxnet0:

$ VBoxManage dhcpserver remove --ifname vboxnet0

Vagrant

If you wish to edit the cluster host names or node IP's you can edit the corresponding entries in the Vagrantfile:

nodes = [
  { :hostname => 'node1', :ip => '10.125.0.11', :box => 'ubuntu1204-chef'},
  { :hostname => 'node2', :ip => '10.125.0.12', :box => 'ubuntu1204-chef'},
  { :hostname => 'node3', :ip => '10.125.0.13', :box => 'ubuntu1204-chef'}
]

You can also change the :box name to use an existing base box or simply add/remove nodes here to grow or shrink your test cluster..

Chef Cookbook

The nodes of your cluster are defined in the cluster_disk_nodes entry of the RabbitMQ cookbook attribute file at ./cookbooks/rabbitmq/attributes/default.rb. You must enable clustering and you may wish to personalise the shared erlang cookie to aid in debugging if later required.

Change from default values of:

#clustering
default['rabbitmq']['cluster'] = false
default['rabbitmq']['cluster_disk_nodes'] = []
default['rabbitmq']['erlang_cookie'] = 'AnyAlphaNumericStringWillDo'

to

#clustering
default['rabbitmq']['cluster'] = true
default['rabbitmq']['cluster_disk_nodes'] = ['rabbit@node1', 'rabbit@node2', 'rabbit@node3']
default['rabbitmq']['erlang_cookie'] = 'AnyAlphaNumericStringWillDoKJR2013071901'

Vagrant Up

We should now be ready to spin up the cluster with the command:

$ vagrant up

This may take some time as it needs to import the boxes and bring up each node in series. All going well it should spin up all the boses defiined in your Vagrantfile, install RabbitMQ and configure clustering with mirrored queues as per the cookbook attributes.

Verification

The cluster status can be checked on each of the cluster nodes:

'$ vagrant ssh node1'

vagrant@node1:~$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit@node1 ...
[{nodes,[{disc,[rabbit@node1,rabbit@node2,rabbit@node3]}]},
 {running_nodes,[rabbit@node3,rabbit@node2,rabbit@node1]},
 {partitions,[]}]
...done.

The vagrant file we're using also installs the management plugin so we should be able to open a browser to the management web UI on port 15672: http://node1:15672/ (you may need to use the IP if hostmanager failed to configure the host hosts.conf file). Log in with guest:guest

RabbitMQ Management Interface

For a quick test to see if queue mirroring is working use the web management interface to create a suitably long lived queue on one node and then use the CLI on a different node to list the queues. The queue created on one node should be mirrored to all other nodes in the cluster. Select "Add a new queue" under the Queues tab of the management web UI:

RabbitMQ Queues

and login to a different cluster node (e.g. vagrant ssh node3) to verify that the queue is listed on that node:

$ vagrant ssh node3

vagrant@node3:~$ sudo rabbitmqctl list_queues
Listing queues ...
MytestQueue 0
...done.

References

RabbitMQ Clustering Guide

RabbitMQ Management

RabbitMQ HA

Notes on Vagrant

For those not familiar with Vagrant the following commands may help:

  • To start all nodes in the vagrant project : $ vagrant up
  • To stop all nodes in the vagrant project : $ vagrant halt
  • To start an individual node in the vagrant project : $ vagrant up <nodeName>
  • To stop an individual node in the vagrant project : $ vagrant halt <nodeName>
  • To log into a vagrant node : $ vagrant ssh <nodeName>

More at the Vagrant docs