Celery workers reserving the task for long time and executing after task expiry

[harshita01398]

Checklist

• I have verified that the issue exists against the master branch of Celery.
• This has already been asked to the discussions forum first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues

• Celery with concurrency does not use all workers while multiple tasks reserved #7277

Possible Duplicates

• Task has been received, but not execute at once. #7795

Environment & Settings

Celery version:
celery==4.3.1
redis==4.1.2
kombu==4.6.11

celery report Output:

Steps to Reproduce

Required Dependencies

• Minimal Python Version: 3.7
• Minimal Celery Version: 4.3.1
• Minimal Kombu Version: 4.6.11
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

Other Dependencies

N/A

Minimally Reproducible Test Case

Expected Behavior

After a task is redelivered to new worker post visibility timeout, the task should be removed from previous worker's reserved list.
If a task is expired, it should not be executed by the worker.

Actual Behavior

A task is received by a worker but not executed. It then gets redelivered to a new worker after visibility timeout (3600). The new worker executes the task. And then, after 2-3 days, original worker also executed the task. (even though task has an expiration of 6 hours)

Logs:
worker32.log:[2022-11-14 14:53:33,063: INFO/MainProcess] Received task: task.version[82b3ff71-e5ec-xxxx-915a-ba102ff51xxx] expires:[2022-11-14 20:53:32.659098+00:00]

[2022-11-18 00:02:33,906: INFO/ForkPoolWorker-732] Task task.version[82b3ff71-e5ec-xxxx-915a-ba102ff51xxx] succeeded in 0.42657385021448135s:

worker33.log:[2022-11-14 15:53:36,000: INFO/MainProcess] Received task: task.version[82b3ff71-e5ec-xxxx-915a-ba102ff51xxx] expires:[2022-11-14 20:53:32.659098+00:00]

[2022-11-14 15:53:37,000: INFO/ForkPoolWorker-64] Task task.version[82b3ff71-e5ec-xxxx-915a-ba102ff51xxx] succeeded in 0.2886648119892925s:

I checked another case currently going on - worker A has reserved a task, it got redelivered to new worker B after 1 hour (visibility timeout). Worker B executed the task. But the task is still reserved with the worker A for more than 2 days and not getting executed.

workerA: [2022-11-18 08:44:06,400: INFO/MainProcess] Received task: api.pkg.get_ticket_info[d7f42e6e-xxxx-471b-ba7d-14a6d666exxx] expires:[2022-11-18 14:44:06.140396+00:00]
user:/var/log/celery$ date
Sun Nov 20 17:36:23 UTC 2022

i.reserved()
{'workerA': [{'id': 'd7f42e6e-xxxx-471b-ba7d-14a6d666exxx', 'name': 'api.pkg.get_ticket_info', 'args': '()', 'kwargs': '{}', 'type': 'oapi.pkg.get_ticket_info', 'hostname': 'workerA', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'hwproxyprod', 'priority': None, 'redelivered': None}, 'worker_pid': None}]}

workerB: [2022-11-18 09:44:07,886: INFO/MainProcess] Received task: api.pkg.get_ticket_info[d7f42e6e-xxxx-471b-ba7d-14a6d666exxx] expires:[2022-11-18 14:44:06.140396+00:00]
[2022-11-18 09:44:09,395: INFO/ForkPoolWorker-776] Task pi.pkg.get_ticket_info[d7f42e6e-xxxx-471b-ba7d-14a6d666exxx] succeeded in 1.293412797152996s: (True, {'url': 'xyz'})

[auvipy]

unfortunately we stopped support for those old versions.

[harshita01398]

@auvipy Can you suggest if moving to latest version would fix the issue. Moving to latest version will require lot of changes from our end and would like to know if these bugs are fixed in latest version.

Because similar issue is posted hre with version 5.2.7 - #7795

[auvipy]

In that case can we close this issue as a duplicate?