[WIP] Feature to avoid hang procs

[schipiga]

Hello @noxdafox ,

As I mentioned some time ago, working with ProcessPool made some headache for me due to hang & zombie processes. And this patch solved it in my case:

• 1st commit about zombies management, and despite of I can't repeat it, I used well-known recommendation http://www.microhowto.info/howto/reap_zombie_processes_using_a_sigchld_handler.html, and looked to python code in gunicorn, project which I intrust to.

• 2nd commit about slept child processes, which can stay after parent process finish. Example below helps to repeat it:

import os
import time

from pebble import ProcessPool


def main():
    pid = os.fork()


pool = ProcessPool(max_workers=1)
with pool:
    pool.schedule(main)
time.sleep(10)

launch:

root@99085a9eeb5f:/# python proba.py&
[1] 1432

observe launched processes:

root@99085a9eeb5f:/# ps aux|grep python
root      1432  1.2  0.1 243732 16228 pts/0    S    09:10   0:00 python proba.py
root      1437  0.0  0.1  22432  9668 pts/0    S    09:10   0:00 python proba.py
root      1439  0.0  0.0   4832   892 pts/0    S+   09:10   0:00 grep python

observe after 10 sec:

root@99085a9eeb5f:/# ps aux|grep python
root      1437  0.0  0.1  22432  9668 pts/0    S    09:10   0:00 python proba.py
root      1441  0.0  0.0   4832   892 pts/0    S+   09:11   0:00 grep python
[1]+  Done                    python proba.py

let's see, what it waits for:

root@99085a9eeb5f:/# strace -p 1437
strace: Process 1437 attached
select(4, [3], [], [], NULL

As I understand, something wrong happens, because it forked after fork, which had lead to descriptors communication problem.

[schipiga]

not ready yet

[noxdafox]

Hello,

Before submitting a PR I would recommend you to open one or more issue tickets where you state the problems and how to reproduce them.

The second example is expected behaviour and not a problem of the Pool. The Pool is responsible of it's own resources and not of the resources created by the workers.

If a worker spawns a process or creates a file, it should take care of removing them itself.