Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

duplicate task in each work. #3270

Open
anan-lee opened this issue Jun 23, 2016 · 31 comments

Comments

@anan-lee
Copy link

commented Jun 23, 2016

celery: 3.1.23 (Cipater)
rabbitmq-server:3.2.4-1

i found some task send to rabbitmq and then celery worker get duplicate task .

function:
def do_backup(name):
....

for example:
server: do_backup.delay("/tmp/__36")

worker02:
[2016-06-23 20:38:27,182: INFO/MainProcess] Received task: tasks.do_backup[40159186-1b88-4d98-aabf-839bb4b7a852]
[2016-06-23 20:38:27,183: INFO/MainProcess] tasks.do_backup[40159186-1b88-4d98-aabf-839bb4b7a852]: get file name: /tmp/__36
worker12:
[2016-06-23 20:36:21,647: INFO/MainProcess] Received task: tasks.do_backup[d265ffba-fa21-48b6-baad-2c0185a3168f]
........
[2016-06-23 20:37:01,492: INFO/MainProcess] tasks.do_backup[d265ffba-fa21-48b6-baad-2c0185a3168f]: get file name: /tmp/__36
[2016-06-23 20:37:01,554: INFO/MainProcess] tasks.do_backup[d265ffba-fa21-48b6-baad-2c0185a3168f]: do task

i make sure the server only send a "/tmp/__36"

@ask

This comment has been minimized.

Copy link
Contributor

commented Jun 28, 2016

They don't seem to be duplicates as they have different task ids?

@anan-lee

This comment has been minimized.

Copy link
Author

commented Jul 5, 2016

thanks for your reply, It is a fact that the same data (/tmp/__36) send to two works . I think the one task don't send ack in time, and so the rabbitmq send the this task to another one?

@DurandA

This comment has been minimized.

Copy link

commented Sep 25, 2016

I have the same issue, I can see on the beat log that the task is only sent once but received twice by the worker a few seconds apart. When using CELERY_ENABLE_UTC=True, the worker receive two tasks, one in UTC time and the other in local time.

I'm using Celery 3.1.23 (Cipater) with Redis 3.2.3.

@cesterlizi

This comment has been minimized.

Copy link

commented Oct 7, 2016

Same Issue, but taskid is the same @ask @DurandA. Celery 3.1.23, Rabbit 3.6.5
I've also added some debug to verify that is ack -> https://github.com/cesterlizi/celery/blob/3daf891f7db10f3f4c6cd89da3c39cb637400b1f/celery/worker/job.py#L532-L534
late ack is enabled

[2016-10-07 12:08:25,875: INFO/MainProcess] Received task: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:25,876: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fa3f9104e60> (args:(u'recrepli.tasks.replicate', u'7c508306-772f-464e-bb7c-103bc1dca784', [u'frogger', u'/home/gc/recordings_tmp/919/call_98277604.wav'], {}, {u'task': u'recrepli.tasks.replicate', u'group': None, u'is_eager': False, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'replicate', u'exchange': u'celery'}, u'args': [u'frogger', u'/home/gc/recordings_tmp/919/call_98277604.wav'], u'headers': {}, u'correlation_id': None, u'hostname': 'replicate_worker@toro', u'kwargs': {}, u'reply_to': None, u'id': u'7c508306-772f-464e-bb7c-103bc1dca784'}) kwargs:{})
[2016-10-07 12:08:25,876: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:25,876: INFO/Worker-8] recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]: Intentando replicar 0/None /home/gc/recordings_tmp/919/call_98277604.wav from AS frogger
[2016-10-07 12:08:25,876: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] pid:26289
[2016-10-07 12:08:25,876: DEBUG/Worker-8] recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]: Colectando /home/gc/recordings_tmp/919/call_98277604.wav from as frogger
......

[2016-10-07 12:08:26,587: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] return '0'
[2016-10-07 12:08:26,587: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:26,587: INFO/MainProcess] Task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] succeeded in 0.710850441828s: 0

[2016-10-07 12:08:27,504: INFO/MainProcess] Received task: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:32,776: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fa3f9104e60> (args:(u'recrepli.tasks.replicate', u'7c508306-772f-464e-bb7c-103bc1dca784', [u'frogger', u'/home/gc/recordings_tmp/919/call_98277604.wav'], {}, {u'task': u'recrepli.tasks.replicate', u'group': None, u'is_eager': False, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'replicate', u'exchange': u'celery'}, u'args': [u'frogger', u'/home/gc/recordings_tmp/919/call_98277604.wav'], u'headers': {}, u'correlation_id': None, u'hostname': 'replicate_worker@toro', u'kwargs': {}, u'reply_to': None, u'id': u'7c508306-772f-464e-bb7c-103bc1dca784'}) kwargs:{})
[2016-10-07 12:08:40,273: INFO/Worker-5] recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]: Intentando replicar 0/None /home/gc/recordings_tmp/919/call_98277604.wav from AS frogger
[2016-10-07 12:08:40,273: DEBUG/Worker-5] recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]: Colectando /home/gc/recordings_tmp/919/call_98277604.wav from as frogger
[2016-10-07 12:08:40,273: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:40,273: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] pid:26286
....

[2016-10-07 12:08:40,286: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] return '1'
[2016-10-07 12:08:40,286: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784]
[2016-10-07 12:08:40,286: INFO/MainProcess] Task recrepli.tasks.replicate[7c508306-772f-464e-bb7c-103bc1dca784] succeeded in 0.0134955914691s: 1

@cesterlizi

This comment has been minimized.

Copy link

commented Oct 7, 2016

@ask I'am also attaching a log where yo can see that the task is duplicated and executed in parallel

[2016-10-07 12:13:25,966: INFO/MainProcess] Received task: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:25,966: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fa3f9104e60> (args:(u'recrepli.tasks.replicate', u'8935556a-f6c5-4e0e-8709-d5958e437e08', [u'pacman', u'/home/gc/recordings_tmp/239/call_98278775.wav'], {}, {u'task': u'recrepli.tasks.replicate', u'group': None, u'is_eager': False, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'replicate', u'exchange': u'celery'}, u'args': [u'pacman', u'/home/gc/recordings_tmp/239/call_98278775.wav'], u'headers': {}, u'correlation_id': None, u'hostname': 'replicate_worker@toro', u'kwargs': {}, u'reply_to': None, u'id': u'8935556a-f6c5-4e0e-8709-d5958e437e08'}) kwargs:{})
[2016-10-07 12:13:25,967: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:25,967: INFO/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Intentando replicar 0/None /home/gc/recordings_tmp/239/call_98278775.wav from AS pacman
[2016-10-07 12:13:25,967: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] pid:26287
[2016-10-07 12:13:25,967: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Colectando /home/gc/recordings_tmp/239/call_98278775.wav from as pacman
[2016-10-07 12:13:26,481: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Colectado /home/gc/recordings_collector/call_239_98278775.wav
[2016-10-07 12:13:26,482: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Convirtiendo a ogg /home/gc/recordings_collector/call_239_98278775.wav
[2016-10-07 12:13:26,754: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Ogg file /home/gc/recordings_collector/call_239_98278775.ogg
[2016-10-07 12:13:26,754: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Intentando replicar a los hosts
[2016-10-07 12:13:27,464: INFO/MainProcess] Received task: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:27,464: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fa3f9104e60> (args:(u'recrepli.tasks.replicate', u'8935556a-f6c5-4e0e-8709-d5958e437e08', [u'pacman', u'/home/gc/recordings_tmp/239/call_98278775.wav'], {}, {u'task': u'recrepli.tasks.replicate', u'group': None, u'is_eager': False, u'delivery_info': {u'priority': None, u'redelivered': False, u'routing_key': u'replicate', u'exchange': u'celery'}, u'args': [u'pacman', u'/home/gc/recordings_tmp/239/call_98278775.wav'], u'headers': {}, u'correlation_id': None, u'hostname': 'replicate_worker@toro', u'kwargs': {}, u'reply_to': None, u'id': u'8935556a-f6c5-4e0e-8709-d5958e437e08'}) kwargs:{})
[2016-10-07 12:13:27,465: DEBUG/MainProcess] on_accepted() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:27,465: DEBUG/MainProcess] Task accepted: recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] pid:26291
[2016-10-07 12:13:27,465: INFO/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Intentando replicar 0/None /home/gc/recordings_tmp/239/call_98278775.wav from AS pacman
[2016-10-07 12:13:27,465: DEBUG/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Colectando /home/gc/recordings_tmp/239/call_98278775.wav from as pacman
[2016-10-07 12:13:27,980: DEBUG/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Colectado /home/gc/recordings_collector/call_239_98278775.wav
[2016-10-07 12:13:27,980: DEBUG/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Convirtiendo a ogg /home/gc/recordings_collector/call_239_98278775.wav
[2016-10-07 12:13:27,986: DEBUG/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Ogg file /home/gc/recordings_collector/call_239_98278775.ogg
[2016-10-07 12:13:27,986: DEBUG/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Intentando replicar a los hosts
[2016-10-07 12:13:28,627: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Replicado /home/gc/recordings_collector/call_239_98278775.ogg to hosts ['toro', 'bermeja', 'marsupial', 'pepe']
[2016-10-07 12:13:28,714: INFO/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Indexado: /home/gc/recordings_collector/call_239_98278775.ogg : {'timestamp': 1475853206, '_rev': u'1-ff292a5338b1a81855f44f1774b10010', 'storage': ['toro', 'bermeja', 'marsupial', 'pepe'], 'doctype': 'recording', 'filename': 'call_98278775.ogg', 'channels': 1, 'audioformat': 'ogg', 'duration': 193.44, '_id': u'98278775', 'tenantid': 239, 'size': 558034}
[2016-10-07 12:13:28,715: DEBUG/Worker-6] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: Enviando a replicar a google: /home/gc/recordings/239/call_98278775.ogg
[2016-10-07 12:13:28,723: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] return '0'
[2016-10-07 12:13:28,723: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]
[2016-10-07 12:13:28,724: INFO/MainProcess] Task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] succeeded in 2.75686794892s: 0
[2016-10-07 12:13:28,910: ERROR/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: [Error 127] 'WebDav: LocalResourceNotFound()'
[2016-10-07 12:13:28,910: WARNING/Worker-10] recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]: No se pudo replicar: /home/gc/recordings_tmp/239/call_98278775.wav, reintentar
[2016-10-07 12:13:28,912: DEBUG/MainProcess] on_sucess() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] return <ExceptionInfo: Retry(u'No se pudo replicar', 120)>
[2016-10-07 12:13:28,912: WARNING/MainProcess] on_success() for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] with ExceptionInfo: <ExceptionInfo: Retry(u'No se pudo replicar', 120)>
[2016-10-07 12:13:28,912: WARNING/MainProcess] on_failure(): for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] ExceptionInfo: <ExceptionInfo: Retry(u'No se pudo replicar', 120)>
[2016-10-07 12:13:28,912: WARNING/MainProcess] on_retry(): for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08] ExceptionInfo: <ExceptionInfo: Retry(u'No se pudo replicar', 120)>
[2016-10-07 12:13:28,912: DEBUG/MainProcess] acknowledge(): for task recrepli.tasks.replicate[8935556a-f6c5-4e0e-8709-d5958e437e08]

@cesterlizi

This comment has been minimized.

Copy link

commented Oct 13, 2016

Hi
I found that duplication is generated when I publish a task. I this case I'am sending the task to rabbit using JAVA rabbitmq API with message confirmation. For some reason (yet not understood) confirmation fails and I am sending the task again and two task message are enqueued to rabbitmq so two tasks are consumed and executed by celery worker

@ask dismiss my assertion in this issue

Carlos

@Skyross

This comment has been minimized.

Copy link

commented Nov 24, 2016

Hi all

We've found same issue in several our projects when we used Redis as broker but seems it's not broker related problem. It's really looks like (we've checked the logs) any delayed task (ETA used, for example with countdown argument) several workers pulls same task (with same Task ID). And I found a lot of posts in Internet that describes that situation but in different words (post, issue). And there is repository specifically for that case. May be that lock should be implemented at Celery level for any broker?

We using this our snippet to prevent such task behavior:

import datetime

from celery import Task
from celery.utils.log import get_task_logger
from django.conf import settings
from django.core.cache import get_cache

logger = get_task_logger(__name__)


# noinspection PyAbstractClass
class TaskWithLock(Task):
    """
    Base task with lock to prevent multiple execution of tasks with ETA.
    It's happens with multiple workers for tasks with any delay (countdown, ETA).
    You may override cache backend by setting `CELERY_TASK_LOCK_CACHE` in your Django settings file
    """
    abstract = True
    lock_expiration = 60 * 60 * 24  # 1 day

    cache = get_cache(getattr(settings, 'CELERY_TASK_LOCK_CACHE', 'default'))

    @property
    def lock_key(self):
        """Unique string for task as lock key"""
        return '%s_%s' % (self.__class__.__name__, self.request.id)

    def acquire_lock(self):
        """Set lock"""
        result = self.cache.add(self.lock_key, True, self.lock_expiration)
        logger.debug('Lock for %s at %s %s', self.request.id, datetime.datetime.now(), 'succeed' if result else 'failed')
        return result

    def release_lock(self):
        """Release lock"""
        result = self.cache.delete(self.lock_key)
        logger.debug('Lock release for %s at %s %s', self.request.id, datetime.datetime.now(), 'succeed' if result else 'failed')
        return result

    def retry(self, *args, **kwargs):
        """We need to release our lock to let the first process take current task in execution on retry"""
        logger.debug('Retry requested %s, %s', args, kwargs)
        self.release_lock()
        return super(TaskWithLock, self).retry(*args, **kwargs)

    def __call__(self, *args, **kwargs):
        """Checking for lock existence"""
        if self.acquire_lock():
            logger.debug('Task %s execution with lock started', self.request.id)
            return super(TaskWithLock, self).__call__(*args, **kwargs)
        logger.debug('Task %s skipped due lock detection', self.request.id)

Feel free to test and point to any errors in this snippet.

@andynguyen11

This comment has been minimized.

Copy link

commented Nov 28, 2016

@Skyross I've been investigating a similar issue this morning as we are using Redis as our broker too. How long are your ETAs set to? We have multiple tasks with ETAs set anywhere in between 1-3 days. I suspect our issue (maybe yours as well) could possibly be this:

http://docs.celeryproject.org/en/latest/getting-started/brokers/redis.html#redis-visibility-timeout

@topalhan

This comment has been minimized.

Copy link

commented Nov 28, 2016

@andynguyen11, I am not familiar with Redis but duplicate tasks happen on other brokers for various reasons (see #2976) and this solution would help.

@Skyross - Thanks for this. I see your solution uses django cache, which works for me but I am thinking if we can find a more generic solution. How about checking the list of existing scheduled tasks, which one can retrieve by using celery.control.inspect().scheduled().

Skip if the task is already scheduled e.g.

scheduled = celery.control.inspect().scheduled()
if scheduled and self.request.id in [item['request']['id'] for sublist in celery.control.inspect().scheduled().values()  for item in sublist]:

Also, are you using this for workflows or single tasks ?

@Skyross

This comment has been minimized.

Copy link

commented Nov 30, 2016

Hi all again,

Previous version of my code contain at least one error: all tasks with retry will be skipped too. I'm testing new version of that class. As you may see now we don't release lock but just waiting it's expiration.

Also I created public repository for that sandbox purpose (may be later I'll move it to GitHub later if needed)

@andynguyen11 Yes, you were right! I came to the same reason. Anyway, sometimes we need to create delay further in future than visibility_timeout value.

@topalhan Feel free to provide your version and compare performance 👍

@auvipy

This comment has been minimized.

Copy link
Member

commented Jan 15, 2018

what's it's situation after 4.1/master?

@Alexeyisme

This comment has been minimized.

Copy link

commented Jan 16, 2018

Hi! I also encountered duplication problem.
We have ETA tasks for 1-4 hours ahead. In the attached log You can see, that task with id 06eaa034-0e0c-49e8-88ec-08279c871cb1 applied twice and executed twice (client device received two same push-notifications).
Already tried huge VISIBILITY_TIMEOUT.

2018-01-15T14:17:01.102524+00:00 app[worker.1]: [2018-01-15 14:17:01,102: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f05a9003598> (args:('notifications.event_update_tasks.send_event_update_notification', '06eaa034-0e0c-49e8-88ec-08279c871cb1', {'lang': 'py', 'task': 'notifications.event_update_tasks.send_event_update_notification', 'id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'eta': '2018-01-15T14:17:00.010508+00:00', 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'parent_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'argsrepr': '(15446, 163)', 'kwargsrepr': '{}', 'origin': 'gen11@e8457c44-4b6c-4d27-b317-9c6b7cc069d0', 'redelivered': True, 'reply_to': '6d7d0c3d-b5d2-3141-850f-6bf23ac825af', 'correlation_id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[15446, 163], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
2018-01-15T14:17:01.102886+00:00 app[worker.1]: [2018-01-15 14:17:01,102: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7f05a9003598> (args:('notifications.event_update_tasks.send_event_update_notification', '06eaa034-0e0c-49e8-88ec-08279c871cb1', {'lang': 'py', 'task': 'notifications.event_update_tasks.send_event_update_notification', 'id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'eta': '2018-01-15T14:17:00.010508+00:00', 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'parent_id': '9dd82f0e-9af3-4714-9ac1-f96b994b5d98', 'argsrepr': '(15446, 163)', 'kwargsrepr': '{}', 'origin': 'gen11@e8457c44-4b6c-4d27-b317-9c6b7cc069d0', 'reply_to': '6d7d0c3d-b5d2-3141-850f-6bf23ac825af', 'correlation_id': '06eaa034-0e0c-49e8-88ec-08279c871cb1', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}}, b'[[15446, 163], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]', 'application/json', 'utf-8') kwargs:{})
2018-01-15T14:17:01.103941+00:00 app[worker.1]: [2018-01-15 14:17:01,103: DEBUG/ForkPoolWorker-1] send_event_update_notification
2018-01-15T14:17:01.104265+00:00 app[worker.1]: [2018-01-15 14:17:01,104: DEBUG/ForkPoolWorker-3] send_event_update_notification
2018-01-15T14:17:01.104976+00:00 app[worker.1]: [2018-01-15 14:17:01,104: DEBUG/MainProcess] Task accepted: notifications.event_update_tasks.send_event_update_notification[255ce589-f8f7-4b7e-9d4f-aba88aa91038] pid:10
2018-01-15T14:17:01.105307+00:00 app[worker.1]: [2018-01-15 14:17:01,105: DEBUG/MainProcess] Task accepted: notifications.event_update_tasks.send_event_update_notification[06eaa034-0e0c-49e8-88ec-08279c871cb1] pid:12
2018-01-15T14:17:01.105661+00:00 app[worker.1]: [2018-01-15 14:17:01,105: DEBUG/MainProcess] Task accepted: notifications.event_update_tasks.send_event_update_notification[06eaa034-0e0c-49e8-88ec-08279c871cb1] pid:13
2018-01-15T14:17:01.106536+00:00 app[worker.1]: [2018-01-15 14:17:01,104: DEBUG/ForkPoolWorker-4] send_event_update_notification
@martin-xia

This comment has been minimized.

Copy link

commented Mar 5, 2018

having the same problem here, duplicate task_id sent to multiple workers within 10 secs. Is there any existing solution?

Moreover, in my situation the task has no countdown.

@tagashy

This comment has been minimized.

Copy link

commented Mar 6, 2018

I'm having the same issue duplicate task with duplicate task on redis ...
i solve it with a decorator (but it's more a hack than something correct) based on https://stackoverflow.com/questions/26831103/avoiding-duplicate-tasks-in-celery-broker

from redis import Redis
from functools import wraps

def run_only_one_instance(lock_name):
    def real_decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            print("run only once " + lock_name + str(args) + str(kwargs))
            try:
                sentinel = Redis(host="redis").incr(lock_name + str(args) + str(kwargs))
                if sentinel == 1:
                    # I am the legitimate running task
                    print("running func")
                    return func(*args, **kwargs)
                else:
                    # Do you want to do something else on task duplicate?
                    print("run is already done escaping")
                Redis(host="redis").decr(lock_name + str(args) + str(kwargs))
            except Exception as e:
                Redis(host="redis").decr(lock_name + str(args) + str(kwargs))
                # potentially log error with Sentry?
                # decrement the counter to insure tasks can run
                # or: raise e

        return wrapper

    return real_decorator

hope it will help some people
in case someone need help to use it it's how i use it:

@app.task
@run_only_one_instance("name_"+"get_all_issues")
def get_all_issues():

EDIT i've forget the return on the wrapper part now it's corrected

@auvipy

This comment has been minimized.

Copy link
Member

commented Mar 6, 2018

can this be added in docs? or any patch in upstream?

@tagashy

This comment has been minimized.

Copy link

commented Mar 6, 2018

If you are speaking about my code
Of course it can be added to docs or patch.
But wait till tomorow i will make a better version whithout hardcoded stuff and i may have found a way to remove the lock name variable and replace it with some data from function
Will do it asap :)

@tagashy

This comment has been minimized.

Copy link

commented Mar 7, 2018

so here is a version that is not hardcoded (but it still work only with redis):
lock_name based on module name + function name + args +kwargs (module name is in the case you got two function with the same name in different module who need to run with the same redis)
redis server address is the only parameter and default to 'redis'
i've also added some exception logging with error from the standard logging lib
and finaly i've removed the print

from redis import Redis
from functools import wraps
from logging import error


def run_only_one_instance(redis_addr: str = "redis"):
    def real_decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            try:
                sentinel = Redis(host=redis_addr).incr(func.__module__+func.__name__ + str(args) + str(kwargs))
                if sentinel == 1:
                    return func(*args, **kwargs)
                else:
                    pass
                Redis(host=redis_addr).decr(func.__module__+func.__name__ + str(args) + str(kwargs))
            except Exception as e:
                Redis(host=redis_addr).decr(func.__module__+func.__name__ + str(args) + str(kwargs))
                error(e)

        return wrapper

    return real_decorator

i've forgot to say it but actualy it work only on python3 (if you want python 2 support remove the ':str' on the definition of run_only_one_instance)

@auvipy

This comment has been minimized.

Copy link
Member

commented Mar 7, 2018

would you mind sending a PR on master?

@tagashy

This comment has been minimized.

Copy link

commented Mar 7, 2018

done but first time I do a PR.
I hope I have done it correctly.
the file is inside celery/utils/

@auvipy auvipy modified the milestones: v5.0.0, v4.3 May 27, 2018

@danield137

This comment has been minimized.

Copy link

commented Jun 5, 2018

@tagashy why would lock based on args,kwargs and not based on task_id?

also, str(args) / str(kwargs) isn't very useful since if it has objects pickled it would fail to compare properly, I would suggest using hash(args)

@tagashy

This comment has been minimized.

Copy link

commented Jun 5, 2018

@danield137

@tagashy why would lock based on args,kwargs and not based on task_id?

How do you access task_id? as my snippet was a wrapper i was thinking of using all the data available to make something like an uuid (not a real one of course) this way if you call foo(bar,toto=toto) and foo(bar,toto) they will not interfere with each other but if you know how to get the task_id it could be great too :)

also, str(args) / str(kwargs) isn't very useful since if it has objects pickled it would fail to compare properly, I would suggest using hash(args)

you are right hash would be better than str
i had discover hash func just recently so i had not think about it when writing this snippet

@Runamok81

This comment has been minimized.

Copy link

commented Jul 11, 2018

New here, just adding a note that we are seeing the same thing. Duplicate work. Running with Celery and Redis. We have high number of duplicate tasks being executed. Celery 4.1.0 and Redis 2.10.6

@andreyrusanov

This comment has been minimized.

Copy link

commented Sep 29, 2018

We have the same issue as @Runamok81 mentioned. Celery 4.1 with Redis.

@auvipy auvipy self-assigned this Nov 18, 2018

@auvipy auvipy modified the milestones: v4.3, v5.0.0 Nov 18, 2018

@viennadd

This comment has been minimized.

Copy link

commented Nov 30, 2018

here is my case, if anyone might need

My celery version:
3.1.25
My situation:
same task id sent to worker twice.
My solution:
clear redis data

I had just renamed the queue,
I had followed the code to kombu.Exchange.DirectExchange.deliver, and I found that celery put a message to the old queue name (not exist in my code anymore) & new queue. Also the worker somehow received both (wired since there is a --queues parameter provided).

I guess Redis was keeping outdated queue config, so I just clear them all, since I won't change the queue name frequently

@metheoryt

This comment has been minimized.

Copy link

commented Mar 13, 2019

Hi, I'm experiencing same problem today - i have many tasks with task_id being duplicated about 40 times, i guess it's a consequence of task being retried many times.

celery 4.1.0, redis

here's the content of queue sync by one task id:

{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:12:25.727044+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "1227b338-5afd-4023-ac86-ea4f673560e4"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:17:11.319382+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen7100@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "127128f4-e4b7-4b60-a6e7-f8b5eba0b282"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:20:49.547095+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "a66d51ab-fa26-4861-b74a-b54344a3ce8b"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:25:36.980849+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen6651@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "623fa9db-53fb-4f02-8a71-e049a18777ae"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:32:56.648995+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen7308@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "7e0da523-2ccb-4d55-b12a-218d11a7881f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:50:30.738049+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "efd0511a-7545-47e2-9748-ac2e57aa1c02"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:54:43.477435+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "2c023d9f-0455-453e-a040-c6c1a12dca0a"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:55:20.960713+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "68190480-dbad-4857-ba6b-cb508b9bb0a0"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T04:57:55.310463+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen7100@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "0ee01ec6-945d-43eb-a21c-e23a64ada8a9"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:00:03.463454+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "e9bc1755-b086-4420-9934-9e8ed767f9a2"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:01:01.604583+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "7f2c116c-70da-45f7-92fb-1133ad6baf2f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:04:31.824229+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "b989b9b0-cdc4-4263-ad4f-7896c4e8ec31"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:07:32.695926+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "54072f13-e20c-4470-9b77-7482b87f8299"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:09:45.829355+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "04c3b6e7-656a-44a5-a377-dd1d08cc4fff"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:17:10.650890+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "df1dd1fa-7e37-42cf-9ce9-2ced878f153e"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:21:16.596645+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "94c82c7c-e0ac-4eaf-b295-ae8cf015625f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:27:23.403676+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "d12b6235-ef1f-4b5e-914c-01a4fc4510a8"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:27:33.804446+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "15f1a689-134e-4b38-873e-427ea5f0a7e9"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:29:47.491450+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "9c0725aa-6309-4b4b-8089-d42645f1b1dd"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:30:43.054268+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "38930a4e-3c3f-4413-ac73-df5a690c64a3"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:30:51.653718+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "ffb0c08d-698e-47d4-996f-c765dae15b6c"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:31:12.896793+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "ffba1969-c220-47fe-a80f-adf2bf8c9447"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:32:54.138908+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "b160b0ff-1184-4d3a-98c8-3cfa24e99f1c"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:33:00.545273+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "e4bb3b01-8a5b-4e0d-b4db-f5f54bc689c7"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:46:50.739562+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "3877e915-e9ec-4a02-a6a5-157c2c630de7"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:48:19.864291+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "c7037ac1-c182-4c86-8a90-2f53f0e5a177"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T05:55:55.877660+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "2eaef446-1a38-492f-a701-6861a8738d08"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:03:22.490942+00:00", "expires": null, "group": null, "retries": 22, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "867e611e-5fcf-401c-a2f5-88c88bec5ad9"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:37:34.111795+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "582067d4-ca18-4cc0-a14d-c110b82a9ccf"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:50:23.583620+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "3002dfd5-a1f3-47ca-bf09-8518783b61df"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:54:30.586274+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "f3727ffd-67c3-43ae-ac98-707ffdf935e8"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T06:55:17.658527+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "bf5463fb-97b0-409b-828f-2d371d91fec5"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:08:37.190744+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "4a4522ba-3c39-43a4-9e6a-8940b0e2db6f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:08:48.661990+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "93ac7edb-0e92-4762-91b7-3e3320085370"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:22:16.080770+00:00", "expires": null, "group": null, "retries": 24, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "13c3f8fd-8866-4291-bb54-2665e4fd3c4f"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:25:35.367521+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17475@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "c64d5b7c-4fad-48ed-be0a-cb4e3e490d23"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:28:03.246552+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "4ea22f97-ab6b-44ed-b84e-a201257fe996"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:30:00.382080+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "da580041-9844-40af-b5de-4d87148a59e4"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:32:17.004536+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17386@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "f43f7bac-76d2-40b9-bed1-c50ed01337c2"}}
{"body": "W1s4MTAwMTI2MTI0ODldLCB7fSwgeyJjYWxsYmFja3MiOiBudWxsLCAiZXJyYmFja3MiOiBudWxsLCAiY2hhaW4iOiBudWxsLCAiY2hvcmQiOiBudWxsfV0=", "content-encoding": "utf-8", "content-type": "application/json", "headers": {"lang": "py", "task": "sync-order-status", "id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "eta": "2019-03-13T07:35:24.480076+00:00", "expires": null, "group": null, "retries": 23, "timelimit": [null, null], "root_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "parent_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "argsrepr": "(810012612489,)", "kwargsrepr": "{}", "origin": "gen17474@ip-10-192-2-170.eu-central-1.compute.internal"}, "properties": {"correlation_id": "b51e9a27-f06b-4692-bb38-98d33e013e0e", "reply_to": "be10bae8-e0f9-30e2-a900-2b6f14a2c577", "origin": "gen17473@ip-10-192-2-170.eu-central-1.compute.internal", "delivery_mode": 2, "delivery_info": {"exchange": "", "routing_key": "sync"}, "priority": 0, "body_encoding": "base64", "delivery_tag": "dd4396c0-6d60-47f3-ae54-e4fe72f8f8ec"}}
@metheoryt

This comment has been minimized.

Copy link

commented Mar 27, 2019

updated to celery 4.3.0rc2, the problem still exists, tasks being duplicated overnight multiple times so 20 task can grow up to 12000. Can you please help me with this issue? which version should I downgrade to avoid this bug?

@thedrow

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

It seems like this is a very old bug. I don't know which version of Celery does not contain it.
In any case, since this happens on Celery 3 and we don't support it anymore I suggest the following:
One of you who can reproduce this should provide a test case or attempt to fix it and send a PR.

@auvipy auvipy removed this from the v5.0.0 milestone Apr 2, 2019

@auvipy auvipy removed their assignment Apr 2, 2019

@auvipy auvipy self-assigned this May 11, 2019

@palmchou

This comment has been minimized.

Copy link

commented May 26, 2019

Providing a DP here. My setting is similar to many others, celery 4.2.1 with redis and the duplicated task is a delayed task with ETA. Multiple workers received the task at the same time.

@auvipy

This comment has been minimized.

Copy link
Member

commented May 26, 2019

any testcase?

@pacoyang

This comment has been minimized.

Copy link

commented Jun 30, 2019

Same issue with redis broker, delay task with ETA future than visibility_timeout will create duplicate task

celery==4.3.0
billiard==3.6.0.0
kombu==4.6.3
pytz==2019.1
redis==3.2.1
vine==1.3.0

switch to rabbitmq solve the problem.

@auvipy auvipy added this to the 4.6 milestone Jul 6, 2019

@adityaseth09

This comment has been minimized.

Copy link

commented Jul 15, 2019

Any update if there is going to be a fix for AWS-SQS soon, as we have a similar problem there too!

@auvipy auvipy modified the milestones: 4.6, 4.5 Jul 16, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.