Mounting volume + RABBITMQ_NODENAME env var crashes the container

[vovimayhem]

I saw this: when mounting a host directory into /var/lib/rabbitmq, setting the "Node Name" with RABBITMQ_NODENAME environment variable crashes the container when re-generated.

# Create a dir to mount to the container:
cd /tmp && mkdir -p my-rabbit-vol

# Create the first container & attach to log:
docker run -d -e RABBITMQ_NODENAME=my-rabbit -v /tmp/my-rabbit-vol:/var/lib/rabbitmq --name my-rabbit rabbitmq:latest && docker logs -f my-rabbit

# Hit CTRL+C and destroy container:
docker stop my-rabbit && docker rm -v -f my-rabbit

# Create again the container with the same node name, and watch it crash:
docker run -d -e RABBITMQ_NODENAME=my-rabbit -v /tmp/my-rabbit-vol:/var/lib/rabbitmq --name my-rabbit rabbitmq:latest && docker logs -f my-rabbit

This behavior is not observed when the node name is set with the --hostname option:

# Hit CTRL+C and destroy the previous example container:
docker stop my-rabbit && docker rm -v -f my-rabbit

# Create the first container & attach to log:
docker run -d --hostname my-rabbit -v /tmp/my-rabbit-vol:/var/lib/rabbitmq --name my-rabbit rabbitmq:latest && docker logs -f my-rabbit

# Hit CTRL+C and destroy container:
docker stop my-rabbit && docker rm -v -f my-rabbit

# Create again the container with the same node name, and watch it go smoothly:
docker run -d --hostname my-rabbit -v /tmp/my-rabbit-vol:/var/lib/rabbitmq --name my-rabbit rabbitmq:latest && docker logs -f my-rabbit

Checking the logs, this is the difference:

Log output extract when --hostname option is set:

=INFO REPORT==== 31-Jan-2015::17:09:47 ===
node           : rabbit@my-rabbit
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : qOtdWaDzvgKk6z1GifpTzw==
log            : undefined
sasl log       : undefined
database dir   : /var/lib/rabbitmq/mnesia/rabbit@my-rabbit # <- Watch this value in the next example

Log output extract when RABBITMQ_NODENAME is set:

=INFO REPORT==== 31-Jan-2015::17:13:03 ===
node           : my-rabbit@1d4ff08db83a
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : qOtdWaDzvgKk6z1GifpTzw==
log            : undefined
sasl log       : undefined
database dir   : /var/lib/rabbitmq/mnesia/my-rabbit

I heavily suspect this is not the expected behavior (please confirm it)

I'll fork & check it out anyways :)

[vovimayhem]

I saw this is actually a RabbitMQ behavior, and not the container behavior:
https://www.rabbitmq.com/man/rabbitmq-env.conf.5.man.html

[vovimayhem]

This is crearly not a problem in the Dockerfile.

Should we put something in the project's README that alerts about this behavior? It is, after all, right there where I found about this functionality!

[tianon]

This is definitely bizarre and shouldn't be happening from what I can tell. I'm not seeing anywhere on that page that says "it's supposed to crash". 👍

[yosifkit]

It seems the error is that it is trying to cluster with the old container. Perhaps there is a way to disable this?

=INFO REPORT==== 2-Feb-2015::16:55:38 ===
node           : my-rabbit@16210be19e7f

...

Error description:
   {boot_step,database,
       {error,
           {failed_to_cluster_with,
               ['my-rabbit@afbf88f6346f'],
               "Mnesia could not connect to any nodes."}}}

[fabiosussetto]

+1, same error here. Setting --hostname worked for me.

[dumbbell]

Hi!

The problem here is that the container's hostname changes between both runs but this hostname is stored by Mnesia (a database provided by Erlang standard library) in /var/lib/rabbitmq/mnesia/...:

• When using --hostname:

  =INFO REPORT==== 31-Jan-2015::17:09:47 ===
  node           : rabbit@my-rabbit
  

• When using RABBITMQ_NODENAME:

  =INFO REPORT==== 31-Jan-2015::17:13:03 ===
  node           : my-rabbit@1d4ff08db83a
  

The default node name is rabbit@$(hostname -s), but can be overriden by $RABBITMQ_NODENAME. If the latter doesn't contain the host part (@...), Erlang automatically adds $(hostname -s). That's why in the first case you end up with rabbit@my-rabbit (the default node name + hostname -s) but in the second case, you get $RABBITMQ_NODENAME (without the host part) + the random hostname generated by Docker.

You can't reuse a Mnesia directory created with one node in another node. So the crash is somewhat expected because Mnesia is lost. If you want to keep /var/lib/mnesia, you need to ensure the node name remains the same. So --hostname is probably the way to go.