#001 - Logic to handle child processes (workers) being killed by the OS (for example, due to Out Of Memory (OOM))

[artpdr]

I'm creating an issue for this change in condu because it might change a vast proportion of code.
The problem is:

• When a worker process (child process that polls tasks from conductor server) is killed by the OS, for example due to OOM, there is no way to update the task being treated by the process that is gone as "FAILED".
• The process that is killed is set as a Zombie process and the task type that was being handled by this process is never polled again from this worker.

The solution that I will provide is:

• When child processes are triggered, they will be triggered from different threads that maintain a pipe open to the process that is triggered.
• Whenever the process receives a task from Conductor server it communicates this task with the thread that triggered the process.
• If the child process is killed, for example due to OOM, the thread that created the process can observe this and can update the task as FAILED and create a new process to replace the one killed before.

For any doubts or suggestions, comment please.

[TheBrunoLopes]

This seems like a viable approach.
There are a few caveats to should be taken into consideration. It's preferable to use new processes instead of new threads because of the GIL. This builtin library should be able to handle the tasks you propose.

[artpdr]

Hi @bioslikk long time no see :p.
As the threads will only be used to have a Pipe for the process they create, I think there is no big overhead in context switching between threads.
PS: I'm aware of GIL.
Cheers.

[artpdr]

It seems that I will need to trigger new processes instead of new threads as launching processes from multi-thread processes seems problematic

[TheBrunoLopes]

Yup, that's what I meant to say. Multiprocessing lib is awesome to tackle that problem.

[artpdr]

Finished this using just processes. There is no multi-threading....
This will be released in Pypi with version 2.0.0b05