Feature request: Specify a timeout after which a worker is stopped

[charalamm]

In case a worker hangs/exceeds some time limit it would be nice to have a way to kill that worker without affecting the other ones.

For example the pdfkit package is known to produce hanging processes so it would be useful to be able to kill workers that happen to end up with such processes.

I can try to implement this feature if you also think it is a good idea.

[sybrenjansen]

I like the idea of a timeout in a worker. The question is if you want a timeout per worker, or a timeout per task. The workers handle multiple tasks, depending on the settings, so a timeout per task sounds like what you're after.

What are your ideas of implementing this?

[ecstorms]

A quick and dirty approach I've used in the past is to automatically wrap any function passed to a worker with a timeout wrapper, and then create a TimeoutError exception when the worker is called to generate a result.

[sybrenjansen]

It's not as easy as it sounds. Most timeout packages I know use signals under the hood, which won't work under Windows. There's the https://pypi.org/project/wrapt-timeout-decorator/ package, which works under all platforms, but creates an additional process under Windows. This is a bit of overkill, because then you would create a process within a process. Or it would defeat the purpose if you want to use threading.

It would be better to utilize the existing workers somehow. You could set a timestamp right before a task is executed and set it back to 0 when the task is done. You can then let the main process check if a worker took to long and terminate if needed. That sounds like the least overhead to me. The downside of this approach, however, is that this won't work in threading, as you can't kill a thread. At least not in a normal way. If anyone knows of a cross-platform solution for killing threads robustly, then I'm all ears.

This downside doesn't have to keep us from implementing it for processes, however. As long as it's documented that threads won't work that nicely with a timeout.

[ecstorms]

So you're thinking of something like having the main thread keep a timer for each worker, which it initializes at 0 whenever it passes a job to a worker and freezes when it collects a result from a worker, and then updating the timestamps when main thread passes through to collect results, killing off any worker process whose timestamp passes a threshold?

[sybrenjansen]

Something like that. I set the current timestamp per worker right before a task is executed. Then, in the main thread I check occasionally if the current timestamp minus timestamp of a worker exceeds the set timeout. If so, terminate is called and a TimeoutError is raised. When a task is completed the worker sets the timestamp to 0, indicating to the main thread that the task has finished (i.e., don't check for timeouts).

So I only have to set the timestamp right before and after a task, meaning minimal overhead.

I'm already pretty much done with the implementation. Only need to refine a few bits and write some tests.