1% of tasks are not restored on warm shutdown, is this avoidable?

[bang-eoh]

Test scenario

We use kubernetes to deploy the worker with specific queue name smart-parking.
We use this loop to continuously send 200 tasks to this queue

from root.celery import debug_task
import time
for x in range(200):
    debug_task.apply_async(countdown=100, queue='smart_parking')
    time.sleep(10)

On other hand, we use another script to continuously kill the pod with warm shutdown, every time a worker pod is kill, another worker pod will be spawned. As I see, when warm shutdown a worker, not-executed tasks will be restored to the queue, and when new worker is spawned, it received the said tasks.

Expected results

200 tasks will be executed

Current result

We tested 2 times.
First round, because the test got stopped by another thread, only 187 tasks are sent and received by celery, only 185 tasks got executed.
Second round, 200 tasks are received, only 198 executed.

Conclusion

There are 1% rate that a tasks will be lost during a warm shutdown.