Visibility timeout set in celery config does not work, celery still using the default 1 hour

[la-luo-unit21]

Checklist

• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues in this issue
  (If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

Here is how we write the configs file:

result_expires = 60
task_acks_late = True  # enables retry on task failure
worker_prefetch_multiplier = 1  # reserving one task at a time
accept_content = ['json']
result_extended = True
broker_transport_options = {
    'queue_order_strategy': 'round_robin',
    'visibility_timeout': 7200  # by seconds, 2 hours
}
task_send_sent_event = True

Here is how we configure the worker:

        celery = Celery(
            __name__,
            broker=config.CELERY_BROKER_URL,
            backend=config.CELERY_RESULT_BACKEND
        )

        os.environ.setdefault(
            'CELERY_CONFIG_MODULE',
            'celeryconfigs.default'
        )
        celery.config_from_envvar('CELERY_CONFIG_MODULE')
        celery.conf.ONCE = {
            'backend': 'celery_once.backends.Redis',
            'settings': {
                'url': config.CELERY_BROKER_URL,
                'default_timeout': 60 * 60
            }
        }

However, tasks running long still get redelivered after 1 hour as the default visibility timeout, not 2 hours as we set. We are following the docs https://docs.celeryq.dev/en/stable/userguide/configuration.html#broker-transport-options What could we possibly miss?

Possible Duplicates

• None

Description

Suggestions

[open-collective-bot]

Hey @la-luo-unit21 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[andyzheung]

do you solve this problem? @la-luo-unit21

[la-luo-unit21]

do you solve this problem? @la-luo-unit21

Hi we are still experiencing this and see if someone can shed some light here

[auvipy]

#7696 is a duplicate

[auvipy]

you are using redis or SQS transports right? and this is happening in 5.3.0b1?

[liuzm7188]

I have the same issue, and a task is still in unacked queue. When I restart celery worker, the task doesn't get redelivered, so it lossed.
celery==5.2.7
redis==4.3.4
kombu=5.2.4

task_reject_on_worker_lost = True
task_acks_late = True
broker_transport_options = {
'visibility_timeout': 1800,
'health_check_interval': 10
}

[srijan-scanova]

Hi there, I'm using celery with SQS and having the same issue with the countdown 30 min in apply_async args. Please, if you could prioritise this bug, then this would be a great help!

[sVathis]

After a lot of testing I found that in order changes in visibility_timeout value to take effect, one must set it in all, celery config, result backend & transport options, as follows :
visibility_timeout = 600
result_backend_transport_options = broker_transport_options = { 'visibility_timeout': 600 }
So the reported behavior might not be a product defect per se, just lack of proper documentation

[auvipy]

After a lot of testing I found that in order changes in visibility_timeout value to take effect, one must set it in all, celery config, result backend & transport options, as follows : visibility_timeout = 600 result_backend_transport_options = broker_transport_options = { 'visibility_timeout': 600 } So the reported behavior might not be a product defect per se, just lack of proper documentation

would you mind sending a PR to improve the documentation?

[pangyuteng]

To those using rabbitmq as broker, please checkout the "Warning" section under "Calling - ETA and Countdown".
https://docs.celeryq.dev/en/stable/userguide/calling.html#eta-and-countdown
Due to rabbitmq version updates, your rabbitmq config may need to be updated to bump the consumer timeout to a very large value, e.g. consumer_timeout = 31622400000.

in ubuntu, you can set this variable in /etc/rabbitmq/rabitmq-env.conf by adding the below line.
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-rabbit consumer_timeout 31622400000"
https://stackoverflow.com/a/73928874/868736