New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compatibility with multiprocessing 'forkserver' start method #264

Open
davejp opened this Issue May 3, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@davejp
Copy link

davejp commented May 3, 2017

I need to use the forkserver (or spawn) start method in my Linux application. For example:

import time
import multiprocessing

class TestProcess(multiprocessing.Process):

    def __init__(self, q):
        super().__init__()
        self._q = q
        self.start()

    def run(self):
        self._q.put("hello")


if __name__ == '__main__':

    multiprocessing.set_start_method("forkserver")    
    q = multiprocessing.Queue()
    TestProcess(q)
    time.sleep(5)
    print(q.get())

When running unfrozen it works fine and the processes show up as below:

1000     11032  2.3  0.3  71640 10244 pts/1    S+   22:39   0:00 python test.py
1000     11033  2.0  0.2  49840  7896 pts/1    S+   22:39   0:00 /home/danderson/testcx/bin/python -R -c from multiprocessing.semaphore_tracker import main;main(6)
1000     11034  2.3  0.2  71344 10012 pts/1    S+   22:39   0:00 /home/danderson/testcx/bin/python -R -c from multiprocessing.forkserver import main; main(6, 8, ['__main__'], **{'sys_path': ['/home/danderson/testcx', '/opt/python-3.4.3/lib/python34.zip', '/opt/python-3.4.3/lib/python3.4', '/opt/python-3.4.3/lib/python3.4/plat-linux', '/opt/python-3.4.3/lib/python3.4/lib-dynload', '/home/danderson/testcx/lib/python3.4/site-packages']})

When frozen it goes into an endless loop (or if using argparse in the script, it will complain and exit). It appears that when setting the start method manually like this it will generate a semaphore tracking process and however many other processes the program creates.

I can get it to run if I parse sys.argv like so:

if __name__ == '__main__':

    if len(sys.argv) == 5 and sys.argv[3] == "-c":
        exec(sys.argv[4])
    else:
        multiprocessing.set_start_method('forkserver')
        q = multiprocessing.Queue()
        TestProcess(q)
        time.sleep(5)
        print(q.get())

Then it show up in the process list as:

1000     11594  2.6  0.3  75836 10508 pts/2    S+   22:46   0:00 ./test
1000     11595  2.3  0.2  55892  7948 pts/2    S+   22:46   0:00 /home/danderson/testcx/dist/test -S -E -c from multiprocessing.semaphore_tracker import main;main(6)
1000     11596  2.6  0.3  75856 10280 pts/2    S+   22:46   0:00 /home/danderson/testcx/dist/test -S -E -c from multiprocessing.forkserver import main; main(6, 8, ['__main__'], **{'sys_path': ['/home/danderson/testcx/dist', '/home/danderson/testcx/dist/lib/python34.zip', '/home/danderson/testcx/dist/lib/python3.4/', '/home/danderson/testcx/dist/lib/python3.4/plat-linux', '/home/danderson/testcx/dist/lib/python3.4/lib-dynload']})

I am not parsing the -S or -E switches here, but it seems to work.

Another option may be to bundle the standard python interpreter somehow with cx_Freeze and set the path to it with multiprocessing.set_executable?

Has anyone come across this or know of any solutions, other than what I have above?

I am running python 3.4.3 and cx_Freeze 5.0.1 on Ubuntu.

Kind Regards,
David Anderson

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment