You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.
I noticed this when running dramatiq and django_dramatiq in docker while rabbitmq is still starting. Would it make sense to implement a retry with exponential backoff?
Traceback (most recent call last):
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 160, in declare_queue
self._declare_queue(queue_name)
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 179, in _declare_queue
return self.channel.queue_declare(queue=queue_name, durable=True, arguments={
File "/opt/everyprojectlist/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 95, in channel
channel = self.state.channel = self.connection.channel()
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 75, in connection
parameters=self.parameters)
File "/opt/project/app/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 374, in __init__
self._process_io_for_connection_setup()
File "/opt/project/app/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 414, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/opt/project/app/venv/lib/python3.6/site-packages/pika/adapters/blocking_connection.py", line 468, in _flush_output
raise exceptions.ConnectionClosed(maybe_exception)
pika.exceptions.ConnectionClosed: Connection to 172.17.0.2:5672 failed: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/opt/project/app/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/opt/project/app/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 341, in execute
django.setup()
File "/opt/project/app/venv/lib/python3.6/site-packages/django/__init__.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/opt/project/app/venv/lib/python3.6/site-packages/django/apps/registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "/opt/project/app/venv/lib/python3.6/site-packages/django/apps/config.py", line 116, in create
mod = import_module(mod_path)
File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 978, in _gcd_import
File "<frozen importlib._bootstrap>", line 961, in _find_and_load
File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/opt/django_dramatiq/django_dramatiq/apps.py", line 3, in <module>
from .tasks import broker # noqa
File "/opt/django_dramatiq/django_dramatiq/tasks.py", line 39, in <module>
def delete_old_tasks(max_task_age=86400):
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/actor.py", line 74, in actor
return decorator(fn)
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/actor.py", line 69, in decorator
priority=priority, broker=broker, options=options,
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/actor.py", line 89, in __init__
self.broker.declare_actor(self)
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/broker.py", line 158, in declare_actor
self.declare_queue(actor.queue_name)
File "/opt/project/app/venv/lib/python3.6/site-packages/dramatiq/brokers/rabbitmq.py", line 174, in declare_queue
del self.consumer
AttributeError: consumer
The text was updated successfully, but these errors were encountered:
Oops! That should say "channel" not "consumer". Normally what would happen here (assuming the attribute was correctly named) is the process would exit with a return code of 3 signalling to the user that it exited due to a connection error. That felt like a more straightforward and generic behaviour than retrying because then the caller could decide what to do (eg: write a bash script to do the retrying, or leave it to kubernetes, etc.). That said, consumers do retry w/ exponential backoff when rabbitmq suddenly goes away so maybe we should do that on startup as well just to be consistent.
I'll fix the AttributeError tonight and then sleep on the question of automatically retrying w/ backoff on startup.
I fixed the attribute error in 0.12.1. I'm going to punt on retrying with backoff on startup for now since something like that is easy to build on top of the return code. Eg:
#!/usr/bin/env bash
delay=1
whiletrue;do
env PYTHONPATH=. dramatiq foo
if [ $?-eq 3 ];thenecho"Connection error encountered on startup. Retrying in $delay second(s)..."
sleep $delay
delay=$((delay *2))elsebreakfidone
I noticed this when running dramatiq and django_dramatiq in docker while rabbitmq is still starting. Would it make sense to implement a retry with exponential backoff?
The text was updated successfully, but these errors were encountered: