install fails at trying to setup rabbitmq users

[alex]

+ sudo rabbitmqctl delete_user zulip
Deleting user "zulip" ...
Error: unable to connect to node rabbit@e611e15f9c9d: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@e611e15f9c9d]

hosts, their running nodes and ports:
- e611e15f9c9d: [{rabbitmqctl7195,42064}]

current node details:
- node name: rabbitmqctl7195@e611e15f9c9d
- home dir: /var/lib/rabbitmq
- cookie hash: xCFdbNsgD1SPwNlmRbq/ZA==

+ true
+ sudo rabbitmqctl delete_user guest
Deleting user "guest" ...
Error: unable to connect to node rabbit@e611e15f9c9d: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@e611e15f9c9d]

hosts, their running nodes and ports:
- e611e15f9c9d: [{rabbitmqctl7240,40403}]

current node details:
- node name: rabbitmqctl7240@e611e15f9c9d
- home dir: /var/lib/rabbitmq
- cookie hash: xCFdbNsgD1SPwNlmRbq/ZA==

+ true
+ sudo rabbitmqctl add_user zulip 58cfc28bf881b28310986a8a5d1306c2fab8af5aff63562df93456b5cce3b0b7
Creating user "zulip" ...
Error: unable to connect to node rabbit@e611e15f9c9d: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@e611e15f9c9d]

hosts, their running nodes and ports:
- e611e15f9c9d: [{rabbitmqctl7286,53989}]

current node details:
- node name: rabbitmqctl7286@e611e15f9c9d
- home dir: /var/lib/rabbitmq
- cookie hash: xCFdbNsgD1SPwNlmRbq/ZA==

This occurs with the following Dockerfile:

FROM ubuntu:trusty

ENV DEBIAN_FRONTEND noninteractive
# Bump when you want to re-run apt-get upgrade
ENV LAST_UPDATE 2

RUN apt-get update
RUN apt-get -y upgrade
RUN apt-get -y install wget

ENV ZULIP_VERSION "1.3.5"

RUN wget https://www.zulip.com/dist/releases/zulip-server-$ZULIP_VERSION.tar.gz
RUN tar -xvf zulip-server-$ZULIP_VERSION.tar.gz
RUN rm zulip-server-$ZULIP_VERSION.tar.gz
RUN mv zulip-server-$ZULIP_VERSION /root/zulip

WORKDIR /root/zulip
RUN ./scripts/setup/install

[timabbott]

Looks like rabbitmq-server isn't started.

timabbott@f2248d2 is what I needed to do in order to get Travis CI (based on docker) installing a development type environment, which may be helpful for debugging (e.g. I did have to run a rabbitmq restart in the middle of the install process there)

[royalaid]

I tired the above fix and I am still having the install fail.

[timabbott]

Try starting epmd explicitly (one of rabbitmq's dependencies) and then starting rabbitmq -- another development mentioned that worked for them in an environment where they were getting this sort of failure...

(Also the .travis.yml is now merged into master)

[royalaid]

Starting epmd with empd -daemon did not change the result.

Error log below:

Deleting user "guest" ...
Error: unable to connect to node rabbit@hook: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@hook]

hosts, their running nodes and ports:
- hook: [{rabbitmqctl29496,33511}]

current node details:
- node name: rabbitmqctl29496@hook
- home dir: /var/lib/rabbitmq
- cookie hash: 3XsN7TD1p6mRQluJafLl+w==

+ true
+ sudo rabbitmqctl add_user zulip 2fb280f8f8c96133f48fcfac951b7fad567c92a15488246096f3ab74864de340
Creating user "zulip" ...
Error: unable to connect to node rabbit@hook: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@hook]

hosts, their running nodes and ports:
- hook: [{rabbitmqctl29541,57316}]

current node details:
- node name: rabbitmqctl29541@hook
- home dir: /var/lib/rabbitmq
- cookie hash: 3XsN7TD1p6mRQluJafLl+w==

[royalaid]

Update, purging and reinstall rabbitmq seems to "fix" the issue. Followed the steps outlined here. It seems to be an issue related to the install process not going through clean on the first try.

[timabbott]

Cool, thanks for posting the fix. I'm not sure I see an easy way to fix this for future users, but we should probably at least document this in a README.prod.md troubleshooting section...

[royalaid]

An option would be to have start/finish detection for the install and if it gets messed up mid-way through to either roll back or purge and start fresh.

[royalaid]

As well when using the default docker image for ubuntu you need to the modify the /usr/sbin/policy-rc.d from exit 101 to exit 0 as per directions here. Note I am not sure of the implications of making this change but it seems to achieve the desire goal.

Note: This also appears to have implications for postgres during the installation as well preventing it from starting.

[marcstreeter]

we had an issue with rabbitmq not working because the hostname was not set in the /etc/hosts file. After updating it, rabbitmq started up fine.

[royalaid]

Just to verify you are setting it to your FQDN, yes?

[marcstreeter]

I was just using whatever hostname and hostname -f provided, however, let me preface my input with the fact that I never got the complete zulip package to work. The fix was based on this thread https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/653405/comments/12