Task was destroyed but it is pending!

[sentry-io]

Sentry Issue: VOIIO-SAM-9

Task was destroyed but it is pending!
task: <Task pending name='Task-4668' coro=<AsyncioListenerRunner.run.<locals>.run_ack_function_asynchronously() done, defined at /app/.heroku/python/lib/python3.11/site-packages/slack_bolt/listener/asyncio_runner.py:111> wait_for=<Future pending cb=[Task.task_wakeup()]>>

[codingjoe]

Graceful shutdown or restarts are an issue. If you have a ongoing inference run, a SIGTERM will leave it hanging. Which can lead to errors, if you have a run that is pending function execution. Users will not be able to add new messages to a thread or start another run for 10 minutes (run timeout).

[amureki]

Graceful shutdown or restarts are an issue. If you have a ongoing inference run, a SIGTERM will leave it hanging. Which can lead to errors, if you have a run that is pending function execution. Users will not be able to add new messages to a thread or start another run for 10 minutes (run timeout).

So, what I could imagine - on a graceful shutdown - we should keep it in the queue and come back to it as soon as the server is back up.
Or what would you expect here?

[codingjoe]

That's a great idea, how do we know if a run is still in the queue, though? Or rather, how do we remember which runs we should poll again? Listing runs for all threads on redis seems like an overkill. We would store runs in a queue and acknowledge them upon completion.