Dramatiq fails if rabbitmq hasn't been started

[rakanalh]

I noticed this when running dramatiq and django_dramatiq in docker while rabbitmq is still starting. Would it make sense to implement a retry with exponential backoff?

Traceback (most recent call last):
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 160, in declare_queue
    self._declare_queue(queue_name)
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 179, in _declare_queue
    return self.channel.queue_declare(queue=queue_name, durable=True, arguments={
  File "/opt/everyprojectlist/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 95, in channel
    channel = self.state.channel = self.connection.channel()
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 75, in connection
    parameters=self.parameters)
  File "/opt/project/app/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 374, in __init__
    self._process_io_for_connection_setup()
  File "/opt/project/app/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 414, in _process_io_for_connection_setup
    self._open_error_result.is_ready)
  File "/opt/project/app/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 468, in _flush_output
    raise exceptions.ConnectionClosed(maybe_exception)
pika.exceptions.ConnectionClosed: Connection to 172.17.0.2:5672 failed: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/project/app/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/opt/project/app/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 341, in execute
    django.setup()
  File "/opt/project/app/venv/lib/python3.6/site-packages/django/__init__.py", line 27, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/opt/project/app/venv/lib/python3.6/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/opt/project/app/venv/lib/python3.6/site-packages/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/opt/django_dramatiq/django_dramatiq/apps.py", line 3, in <module>
    from .tasks import broker  # noqa
  File "/opt/django_dramatiq/django_dramatiq/tasks.py", line 39, in <module>
    def delete_old_tasks(max_task_age=86400):
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/actor.py", line 74, in actor
    return decorator(fn)
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/actor.py", line 69, in decorator
    priority=priority, broker=broker, options=options,
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/actor.py", line 89, in __init__
    self.broker.declare_actor(self)
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/broker.py", line 158, in declare_actor
    self.declare_queue(actor.queue_name)
  File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 174, in declare_queue
    del self.consumer
AttributeError: consumer

[Bogdanp]

Oops! That should say "channel" not "consumer". Normally what would happen here (assuming the attribute was correctly named) is the process would exit with a return code of 3 signalling to the user that it exited due to a connection error. That felt like a more straightforward and generic behaviour than retrying because then the caller could decide what to do (eg: write a bash script to do the retrying, or leave it to kubernetes, etc.). That said, consumers do retry w/ exponential backoff when rabbitmq suddenly goes away so maybe we should do that on startup as well just to be consistent.

I'll fix the AttributeError tonight and then sleep on the question of automatically retrying w/ backoff on startup.

Thanks!

[Bogdanp]

I fixed the attribute error in 0.12.1. I'm going to punt on retrying with backoff on startup for now since something like that is easy to build on top of the return code. Eg:

#!/usr/bin/env bash

delay=1
while true; do
    env PYTHONPATH=. dramatiq foo
    if [ $? -eq 3 ]; then
        echo "Connection error encountered on startup. Retrying in $delay second(s)..."
        sleep $delay
        delay=$((delay * 2))
    else
        break
    fi
done

[Bogdanp]

I added that as an example to the cookbook: https://dramatiq.io/cookbook.html#retrying-connection-errors-on-startup

Thanks again!