[BUG] Web: RabbitMQ Update sabatoges `acks_late`

[subdavis]

Describe the bug
Jobs longer than 30 minutes are failing after rabbitMq made a behavior change. It's not clear why this just started, perhaps our Rabbit node was updated by AWS without them telling us.

To Reproduce
Using version > 3.8.15, try to run a job for longer than 30 minutes.

Related data

[2021-10-14 06:31:46,642: INFO/MainProcess] Received task: dive_tasks.tasks.run_pipeline[a9bc43bf-47d3-47a7-9eed-b05d9c5c40a2]  
[2021-10-14 07:01:46,699: CRITICAL/MainProcess] Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')

https://groups.google.com/g/celery-users/c/5dPhSfF5APQ

Expected behavior
Jobs should be allowed to run for longer than 30 min.

[waxlamp]

This AWS forum thread implies that the ability to adjust the acknowledgment timeout will be here soon, and that in the meantime people should just use 3.8.11. If that is true, then this would be a good solution for us. If not, we can make a decision on how to proceed when they retire 3.8.11, leaving us in the cold.