Long ETA tasks executed multiple times on Redis broker despite setting visibility_timeout

[vrastogi]

I have the same issue as that in http://stackoverflow.com/q/12683546/567555

Like other commenters to that question, I can confirm that this bug is present. I am using celery 3.1.17, kombu 3.0.24, Redis server 2.8.4 with both the Redis server and the workers running on the same machine. My tasks have a countdown of 6 hours, so I set the visibility_timeout to 43200 (12 hours). Printing the visibility_timeout just before it is used in restore_visible of kombu/transport/redis.py indicates that it is set correctly. But I can still see the same tasks executed multiple times.

Here is my celery configuration, if it matters.

app.config_from_object({
    'BROKER_URL': 'redis://localhost:6379',
    'CELERY_RESULT_BACKEND': 'redis://localhost:6379',
    'CELERYD_POOL_RESTARTS': True,  # Required for /worker/pool/restart API
    'CELERYD_CONCURRENCY': 100,
    # need visibility timeout greater than 6 hours
    'BROKER_TRANSPORT_OPTIONS': {'visibility_timeout': 43200}, # 12 hours
    'CELERYD_TASK_TIME_LIMIT': 1200,
    'CELERY_ACCEPT_CONTENT': ['json'],
    'CELERY_TASK_SERIALIZER': 'json',
    'CELERY_RESULT_SERIALIZER': 'json'
})

Will be happy to provide further information.

[fatihsucu]

Yes this bug still exist. I couldn't override that visibility timeout settings too. I use rabbitmq broker instead of redis broker for solve this problem. If you don't have wait for update i suggest that.

[fatihsucu]

Yes this bug still exist. I couldn't override that visibility timeout settings too. I use rabbitmq broker instead of redis broker for solve this problem. If you don't have wait for update i suggest that.

[vrastogi]

Thanks for another confirmation. I will consider your work-around.

[vrastogi]

Thanks for another confirmation. I will consider your work-around.

[ask]

Just confirmed here, and the visibility timeout is indeed set by BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 30}.

Are you sure you do not have some other consumer running using a different visibility timeout?
E.g. a flower instance running with a default visibility timeout.

Any message consumer using the kombu redis transport will happily restore visible messages
for you, so you have to make sure all of them are using the same configuration.

[ask]

Just confirmed here, and the visibility timeout is indeed set by BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 30}.

Are you sure you do not have some other consumer running using a different visibility timeout?
E.g. a flower instance running with a default visibility timeout.

Any message consumer using the kombu redis transport will happily restore visible messages
for you, so you have to make sure all of them are using the same configuration.

[vrastogi]

Sorry for the late reply. I have not checked anything else, but yes I am running a flower instance. That seems to explain this problem. Is there a way to set a visibility_timeout of flower?

[vrastogi]

Sorry for the late reply. I have not checked anything else, but yes I am running a flower instance. That seems to explain this problem. Is there a way to set a visibility_timeout of flower?

[ask]

Just point it to an app: celery -A proj flower

[ask]

Just point it to an app: celery -A proj flower

[vrastogi]

That is what I have been doing but seems I never restarted flower after changing the visibility_timeout. Closing this issue now.

[vrastogi]

That is what I have been doing but seems I never restarted flower after changing the visibility_timeout. Closing this issue now.