[POC] Requeue tasks after ETA is met

[emarbo]

Fixes #6500

This PR is just a proof of concept as requested here. It is far away to be something complete but it illustrates the idea and addresses the problem.

Description

Workers using --prefetch-multiplier 1 and acks_late should never reserve more tasks than their concurrency level. In other words, they should reserve 1 task for every child worker.

However, when a worker receives an ETA task, it holds the task in memory (unacknowledged) and increases the QoS to allow RabbitMQ to send a new task. So, the QoS of workers is:

worker_qos = prefetch_multiplier * concurrency + tasks_with_future_eta

When the ETA is met, the worker decreases the QoS but still holds the task until one child worker gets idle. This PR attempts to solve the problem by just queueing the task back to RabbitMQ at the same position it was so that other idle consumers can process it immediately.

POC

The eta_poc example has the instructions to test the behaviour and to compare it with the current implementation.

[lgtm-com]

This pull request introduces 1 alert when merging e66b7fb into 239ed3c - view on LGTM.com

new alerts:

• 1 for Module is imported with 'import' and 'import from'

[auvipy]

can you resume the effort to include it in celery 5.4.0 please?