RabbitMQ service fails to stop on Ubuntu 16.04 (Erlang R16B03) with systemd

[rpadovani]

Hello all,
first of all, congrats for your awesome works :)

I'm experiencing issues stopping rabbitmq server using the /etc/init.d/rabbitmq-server stop command.
Taking a look to the logs, I noticed this line:

initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

Now, the problem is Ubuntu doesn't use upstart anymore, but it switched to systemd a couple of releases ago.

After /etc/init.d/rabbitmq-server stop the status of the service is:

rabbitmq-server.service - LSB: Manages RabbitMQ server
   Loaded: loaded (/etc/init.d/rabbitmq-server; bad; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2016-01-20 21:11:40 CET; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11210 ExecStop=/etc/init.d/rabbitmq-server stop (code=exited, status=1/FAILURE)
  Process: 10838 ExecStart=/etc/init.d/rabbitmq-server start (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 512)

Jan 20 21:11:39 artemis su[11295]: + ??? root:rabbitmq
Jan 20 21:11:39 artemis su[11295]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Jan 20 21:11:39 artemis su[11443]: Successful su for rabbitmq by root
Jan 20 21:11:39 artemis su[11443]: + ??? root:rabbitmq
Jan 20 21:11:39 artemis su[11443]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Jan 20 21:11:40 artemis rabbitmq-server[11210]: initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
Jan 20 21:11:40 artemis systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=1
Jan 20 21:11:40 artemis systemd[1]: Stopped LSB: Manages RabbitMQ server.
Jan 20 21:11:40 artemis systemd[1]: rabbitmq-server.service: Unit entered failed state.
Jan 20 21:11:40 artemis systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.

When I start it again, the result is:

● rabbitmq-server.service - LSB: Manages RabbitMQ server
   Loaded: loaded (/etc/init.d/rabbitmq-server; bad; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2016-01-20 21:10:56 CET; 30s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 8565 ExecStop=/etc/init.d/rabbitmq-server stop (code=exited, status=1/FAILURE)
  Process: 9816 ExecStart=/etc/init.d/rabbitmq-server start (code=exited, status=1/FAILURE)
    Tasks: 79 (limit: 512)
   CGroup: /system.slice/rabbitmq-server.service
           ├─10347 /usr/lib/erlang/erts-7.2/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.0...
           ├─10451 inet_gethost 4
           └─10452 inet_gethost 4

Jan 20 21:10:54 artemis su[9898]: + ??? root:rabbitmq
Jan 20 21:10:54 artemis su[9898]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Jan 20 21:10:55 artemis su[10069]: Successful su for rabbitmq by root
Jan 20 21:10:55 artemis su[10069]: + ??? root:rabbitmq
Jan 20 21:10:55 artemis su[10069]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
Jan 20 21:10:56 artemis rabbitmq-server[9816]: initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused
Jan 20 21:10:56 artemis systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=1
Jan 20 21:10:56 artemis systemd[1]: Failed to start LSB: Manages RabbitMQ server.
Jan 20 21:10:56 artemis systemd[1]: rabbitmq-server.service: Unit entered failed state.
Jan 20 21:10:56 artemis systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.

I have to use killall and then start the server again

This is the version of rabbitmq I have:

dpkg -s rabbitmq-server           
Package: rabbitmq-server
Status: install ok installed
Priority: extra
Section: net
Installed-Size: 5569
Maintainer: RabbitMQ Team <info@rabbitmq.com>
Architecture: all
Version: 3.6.0-1
Depends: erlang-nox (>= 1:16.b.3) | esl-erlang, adduser, logrotate
Conffiles:
 /etc/logrotate.d/rabbitmq-server ddaa633357e5f6ff9d7970c6d0a9acd8
 /etc/default/rabbitmq-server 7cc5216f193f6be1cb32c45021197b39
 /etc/init.d/rabbitmq-server e29aacba2db60f7933ace5a2f0e138aa
Description: Multi-protocol messaging broker
 RabbitMQ is an open source multi-protocol messaging broker.
Homepage: http://www.rabbitmq.com/

Let me know if I can help you debugging this :)

[michaelklishin]

What is in RabbitMQ log files?

[rpadovani]

I used tail to checking all files in /var/log/rabbitmq/

The only two with some outputs are rabbit@artemis.log:

=INFO REPORT==== 20-Jan-2016::23:54:46 ===
Stopping RabbitMQ

=INFO REPORT==== 20-Jan-2016::23:54:46 ===
stopped TCP Listener on [::]:5672

=INFO REPORT==== 20-Jan-2016::23:54:47 ===
Stopped RabbitMQ application

=INFO REPORT==== 20-Jan-2016::23:54:47 ===
Halting Erlang VM

and /var/log/rabbitmq/shutdown_log:

Stopping and halting node rabbit@artemis ...

Maybe I need to change a conf to have more output?

[michaelklishin]

No, this means that as far as RabbitMQ is concerned, it has shut down. It's the service script that had an error.

It can be a real struggle to reproduce, e.g. as #76 and #92 have been.

[michaelklishin]

I now think this can be caused by the incorrect PID being passed to systemd, see #664, #573, #666.

[ejogee]

Hi

@michaelklishin I dont think this is a duplicate. The problem relates specifically to the current versions of ubuntu.

@rpadovani How did you manage to get rabbitmq installed on ubuntu 16.04. Installation fails as rabbitMQ attempts to use upstart but Ubuntu has switched to systemd.

If you see the error that you you have on stopping rabbitMQ it is the same error i get on installation. RabbitMQ script is attempting to use upstart but upstart is no longer available on ubuntu by default.

See error below:

systemctl status rabbitmq-server.service
● rabbitmq-server.service - LSB: Manages RabbitMQ server
Loaded: loaded (/etc/init.d/rabbitmq-server; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2016-05-20 10:50:10 SAST; 6min ago
Docs: man:systemd-sysv-generator(8)
Process: 11725 ExecStart=/etc/init.d/rabbitmq-server start (code=exited, status=1/FAILURE)
Tasks: 76 (limit: 512)
CGroup: /system.slice/rabbitmq-server.service
├─11944 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
├─12223 /usr/lib/erlang/erts-7.3/bin/beam.smp -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl --
├─12364 inet_gethost 4
└─12365 inet_gethost 4

May 20 10:50:08 ebrahim-VirtualBox su[11807]: + ??? root:rabbitmq
May 20 10:50:08 ebrahim-VirtualBox su[11807]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
May 20 10:50:08 ebrahim-VirtualBox su[11976]: Successful su for rabbitmq by root
May 20 10:50:08 ebrahim-VirtualBox su[11976]: + ??? root:rabbitmq
May 20 10:50:08 ebrahim-VirtualBox su[11976]: pam_unix(su:session): session opened for user rabbitmq by (uid=0)
May 20 10:50:10 ebrahim-VirtualBox rabbitmq-server[11725]: initctl: Unable to connect to Upstart: Failed to connect to socket /com/ub
May 20 10:50:10 ebrahim-VirtualBox systemd[1]: rabbitmq-server.service: Control process exited, code=exited status=1
May 20 10:50:10 ebrahim-VirtualBox systemd[1]: Failed to start LSB: Manages RabbitMQ server.
May 20 10:50:10 ebrahim-VirtualBox systemd[1]: rabbitmq-server.service: Unit entered failed state.
May 20 10:50:10 ebrahim-VirtualBox systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.

[michaelklishin]

@ejogee what RabbitMQ package version do you use?

[rpadovani]

I use the deb http://www.rabbitmq.com/debian/ testing main PPA, and I installed it on wily with sudo apt-get install rabbitmq-server when wily was in development.

I upgraded to xenial back in November. I do not remember if I had trouble installing it.

[ejogee]

@michaelklishin Im installing via the deb package rabbitmq-server_3.6.2-1_all.deb

[luckydogchina]

Can the question be solved?

[dumbbell]

Hi @luckydogchina! What do you mean?

[michaelklishin]

Our team uses rabbitmq-users for questions.