Are exceptions in _maintain_queue unrecoverable?

[ZadenRB]

I'm running chancy as part of a FastAPI application. Due to transient network issues or DB issues, I've seen errors occur in _maintain_queue or _maintain_queues a couple of times. Root cause is typically being a bad connection in the pool or a connection being killed for exceeding the idle_in_transaction timeout (some psycopg exception being thrown essentially). Whatever the cause, my concern is what happens next. An exception like this is logged:

Task exception was never retrieved
future: <Task finished name='queue_name' coro=<Worker._maintain_queue() done, defined at /usr/local/lib/python3.11/site-packages/chancy/worker.py:320> exception=IdleInTransactionSessionTimeout('terminating connection due to idle-in-transaction timeout')>

And the worker stops picking up jobs from that queue entirely. I have to manually restart the process in order to begin processing jobs again. I'd imagine the same thing happening in maintain_queues would lead to a similar issue, where any queue updates would not be picked up by the worker.

I really have two questions:

1. Does the failure case I've outlined here track with what you would expect? I'm 100% certain I've seen jobs stuck in the queue for 20+ minutes after an error like the one above, but I'm not so familiar with the Chancy codebase, so would love to be corrected if I'm wrong about the root cause.
2. Is this behavior something you would consider changing / adding additional exception handling to? Or am I better off either forking or subclassing the worker (which I'll likely do in the meantime).

[TkTech]

Thanks for the report - for clarification, are you using workers embedded inside of the FastAPI lifecycle system or in standalone workers?

An error like this should be recoverable. We should add faulty connection tests to iron out the behavior as well, as that's currently a gap in coverage. In our deployments we intentionally let the process die fast so ECS will bring up fresh tasks, but requiring a supervisor for recovery shouldn't be mandatory for regular deployments.

[ZadenRB]

The workers are embedded inside the FastAPI lifecycle system. If they were standalone we'd take the approach you mention of allowing a fast crash. But at the scale I'm dealing with running the workers as a separate deployment doesn't make a ton of sense unfortunately.