Redis with Celery, processing fails with unrecoverable error at random times

[PeriGK]

Hi,

I am running celery with redis to schedule some tasks in the background.

Things run fine in general but after a few hours of the bootstrapping I am getting the following error

[2019-10-01 20:56:41,930: CRITICAL/MainProcess] Unrecoverable error: ResponseError('UNBLOCKED force unblock from blocking operation, instance state changed (master -> replica?)') Traceback (most recent call last): File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/worker/worker.py", line 205, in start self.blueprint.start(self) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start return self.obj.start() File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start blueprint.start(self) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start step.start(parent) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 596, in start c.loop(*c.loop_args()) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/celery/worker/loops.py", line 91, in asynloop next(loop) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 362, in create_loop cb(*cbargs) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/kombu/transport/redis.py", line 1070, in on_readable self.cycle.on_readable(fileno) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/kombu/transport/redis.py", line 348, in on_readable chan.handlers[type]() File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/kombu/transport/redis.py", line 726, in _brpop_read **options) File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/redis/client.py", line 853, in parse_response response = connection.read_response() File "/root/.local/share/virtualenvs/mt-scheduler-9rtGwvVU/lib/python3.7/site-packages/redis/connection.py", line 717, in read_response
All tasks do external API invocation.

After that, all the scheduling goes off. Any idea why it is happening, how to prevent it and how to resolve it?

I am using Celery 4.3 and redis library 3.3.8, python 3.7.4. Let me know if you need any additional info.

Thanks,
Periklis

[itamarhaber]

Hello @PeriGK

I'm taking a wild guess here - your Redis server isn't password protected and is open to the public. If this is the case, a nefarious entity is attempting to hijack the server via the SLAVEOF command.

[PeriGK]

Oh that sounds bad. Yes, I am still in early testing and thus there is no much protection around. I have tried with other worker pool types of celery and I will update if needed

Thanks for helping out.

[PeriGK]

Looks like I got the info I wanted. Thanks again @itamarhaber

[itamarhaber]

@PeriGK my pleasure :)