Application doesn't work because of AMQP issue

[myshov]

I just wondered around in the application. And it started to say that couldn't connect to the server. At the time cpu usage jump up to 100%. The reloading of the page give only 500 error page. Error log ful of meesages of this kind:

2015-09-26 13:06:47,982 ERROR    Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/decorator.py", line 85, in wrapper
    return func(request, *args, **kwargs)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/decorator.py", line 474, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/decorator.py", line 301, in _wrapped_view_func
    return authenticate_log_and_execute_json(request, view_func, *args, **kwargs)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/decorator.py", line 288, in authenticate_log_and_execute_json
    process_client(request, user_profile, True)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/decorator.py", line 134, in process_client
    update_user_activity(request, user_profile)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/decorator.py", line 69, in update_user_activity
    queue_json_publish("user_activity", event, lambda event: None)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/lib/queue.py", line 259, in queue_json_publish
    get_queue_client().json_publish(queue_name, event)
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/lib/queue.py", line 240, in get_queue_client
    queue_client = SimpleQueueClient()
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/lib/queue.py", line 25, in __init__
    self._connect()
  File "/home/zulip/deployments/2015-09-26-19-22-05/zerver/lib/queue.py", line 29, in _connect
    self.connection = pika.BlockingConnection(self._get_parameters())
  File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 107, in __init__
    super(BlockingConnection, self).__init__(parameters, None, False)
  File "/usr/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 62, in __init__
    on_close_callback)
  File "/usr/lib/python2.7/dist-packages/pika/connection.py", line 590, in __init__
    self.connect()
  File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 206, in connect
    if not self._adapter_connect():
  File "/usr/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 275, in _adapter_connect
    raise exceptions.AMQPConnectionError(1)
AMQPConnectionError: 1

The application runs on the clean Ubuntu 14.04 on VM with 2Gb and 2 cpu cores. For installation I followed instructions from here https://zulip.org/server.html

ps output:

root@ubuntu:/var/log/zulip# ps aux | sort -rk 3,3 | head -n 20
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
zulip     2109  9.7  4.9 225748 50168 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue user_activity
zulip     2119  9.2  4.7 221452 47972 ?        S    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue message_sender 2
zulip     2095  9.2  5.0 226124 50792 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue slow_queries
zulip     2088  9.2  5.0 226128 50796 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue message_sender 1
zulip     2144  9.1  4.2 192272 42892 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue invites
zulip     2269  9.0  0.9  44264 10088 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py deliver_email
zulip     2224  9.0  2.2 136548 23308 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue feedback_messages
zulip     2148  9.0  4.1 191760 42564 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue missedmessage_mobile_notifications
zulip     2121  8.6  4.5 220012 46476 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue digest_emails
zulip     2263  8.0  1.0  44264 10176 ?        S    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue error_reports
zulip     2257  8.0  1.0  44264 10180 ?        S    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue user_activity_interval
zulip     2190  8.0  3.1 166184 32168 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py runtornado localhost:9993
zulip     2213  7.8  2.6 149264 27232 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue signups
zulip     2248  7.3  1.5 125404 16160 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue message_sender 0
zulip     2305  7.0  1.2 118128 12544 ?        R    23:23   0:00 python /home/zulip/deployments/current/bin/log-management-command /home/zulip/deployments/current/manage.py process_queue user_activity_interval
zulip     2252  7.0  1.4 122092 14776 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue email_mirror
zulip     2240  6.7  2.0 134140 20768 ?        R    23:23   0:00 python /home/zulip/deployments/current/manage.py process_queue missedmessage_emails
root        97 19.5  0.0      0     0 ?        S    20:05  38:45 [kswapd0]
paralle+ 32502  1.6  2.1 335860 21952 ?        Sl   23:21   0:02 /usr/bin/python3 /usr/share/oneconf/oneconf-service

[timabbott]

Looks like a clear case of the RabbitMQ server it depends on either isn't running or isn't configured properly.

You should have a process with "beam.smp" in its name in your full ps output if rabbitmq is running; if it's not, you should debug that (check rabbitmq-server is installed, service restart it, etc.).

Otherwise try running scripts/setup/configure-rabbitmq

[myshov]

@timabbott yes after restarting rabbitmq everything is working now
For some reasons rabbitmq was crashed or killed or I don't know. In the log there are nothing:

=INFO REPORT==== 26-Sep-2015::22:50:18 ===
accepting AMQP connection <0.1905.0> (127.0.0.1:42031 -> 127.0.0.1:5672)

=INFO REPORT==== 27-Sep-2015::15:53:41 ===
Starting RabbitMQ 3.2.4 on Erlang R16B03
Copyright (C) 2007-2013 GoPivotal, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 27-Sep-2015::15:53:41 ===
node           : rabbit@ubuntu
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config
cookie hash    : 4jnxrW5/B/MI3gsF51Oesw==
log            : /var/log/rabbitmq/rabbit@ubuntu.log
sasl log       : /var/log/rabbitmq/rabbit@ubuntu-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@ubuntu

=INFO REPORT==== 27-Sep-2015::15:53:42 ===
Limiting to approx 924 file handles (829 sockets)

As you can see above in the log there was the last connection yesterday and after that rabbitmq crashed or something. And after I restarted it and now it's working. By the way there are no errors in the other logs of rabbitmq.

[timabbott]

Yeah rabbitmq is not great about providing verbose logging information. If it crashed due to an OOM kill or something, there could be something in dmesg/syslog/etc., so maybe worth checking those to see if there's any evidence for what happened.

[SystemZ]

In my case rabbitmq-server never starts

service rabbitmq-server restart
 * Restarting message broker rabbitmq-server   [fail] 

I tried more fresh build of erlang with

echo "deb http://packages.erlang-solutions.com/ubuntu trusty contrib"  > /etc/apt/sources.list.d/erlang.list
apt-get update
apt-get install erlang

But it didn't help.
I'm getting

ERROR: epmd error for host "vps124": address (cannot connect to host/port)

in /var/log/rabbitmq/startup_err

[timabbott]

Try purging and reinstalling the rabbitmq-server package, following these instructions:

http://www.techsfo.com/blog/2013/06/rabbitmq-breaks-when-you-rename-hostname/

Please report back whether that works!

[0ptimus]

I believe that I am having similar issues to this ticket. Getting "500: server error" when clicking link from email to activate new account I checked my log (/var/log/rabbitmq/rabbit@zulip.log) and I see a lot of these errors...

=ERROR REPORT==== 29-Sep-2015::13:23:35 ===
closing AMQP connection <0.520.0> ([::1]:43088 -> [::1]:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "PLAIN login refused: user 'zulip' - invalid credentials",
                             'connection.start_ok'}}

=ERROR REPORT==== 29-Sep-2015::13:23:37 ===
closing AMQP connection <0.524.0> ([::1]:43089 -> [::1]:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "PLAIN login refused: user 'zulip' - invalid credentials",
                             'connection.start_ok'}}

I am not familiar with troubleshooting rabbitmq, can anyone provide some tips to fix this? Want to try out Zulip for my IT dept.

[timabbott]

This issue is most likely that somehow the password Zulip is using doesn't
match the password rabbitmq has configured --
scripts/setup/configure-rabbitmq should fix this.

I should add you'll want to restart the Zulip server via supervisorctl restart all afterwards

[SystemZ]

Thx for link @timabbott ! This solves problem for me :) Zulip works even after vps reboot, finally 😄

service supervisor stop
dpkg -P rabbitmq-server
apt-get install rabbitmq-server
/root/zulip/scripts/setup/configure-rabbitmq
service supervisor start

I'm stopping supervisor because bad AMQP restarts workers every second, causing 100% CPU.

It looks like installation of rabbit-mq goes bad in zulip setup script or maybe it's my env (OpenVZ container).

[timabbott]

rabbitmq-server installation seems to have problems in environments with weird networking configurations; so yeah this probably is an OpenVZ container problem...

[jmandel]

@SystemZ thanks -- I needed to follow your purge-then-reinstall suggestion on a fresh Ubuntu 14.04 install (following the default instructions from https://zulip.org/server.html)

[3ruce]

Yep, this also worked for me after a reboot...

service supervisor stop
dpkg -P rabbitmq-server
apt-get install rabbitmq-server
/root/zulip/scripts/setup/configure-rabbitmq
service supervisor start