compatibility with multiprocessing 'forkserver' start method

[davejp]

I need to use the forkserver (or spawn) start method in my Linux application. For example:

import time
import multiprocessing

class TestProcess(multiprocessing.Process):

    def __init__(self, q):
        super().__init__()
        self._q = q
        self.start()

    def run(self):
        self._q.put("hello")


if __name__ == '__main__':

    multiprocessing.set_start_method("forkserver")    
    q = multiprocessing.Queue()
    TestProcess(q)
    time.sleep(5)
    print(q.get())

When running unfrozen it works fine and the processes show up as below:

1000     11032  2.3  0.3  71640 10244 pts/1    S+   22:39   0:00 python test.py
1000     11033  2.0  0.2  49840  7896 pts/1    S+   22:39   0:00 /home/danderson/testcx/bin/python -R -c from multiprocessing.semaphore_tracker import main;main(6)
1000     11034  2.3  0.2  71344 10012 pts/1    S+   22:39   0:00 /home/danderson/testcx/bin/python -R -c from multiprocessing.forkserver import main; main(6, 8, ['__main__'], **{'sys_path': ['/home/danderson/testcx', '/opt/python-3.4.3/lib/python34.zip', '/opt/python-3.4.3/lib/python3.4', '/opt/python-3.4.3/lib/python3.4/plat-linux', '/opt/python-3.4.3/lib/python3.4/lib-dynload', '/home/danderson/testcx/lib/python3.4/site-packages']})

When frozen it goes into an endless loop (or if using argparse in the script, it will complain and exit). It appears that when setting the start method manually like this it will generate a semaphore tracking process and however many other processes the program creates.

I can get it to run if I parse sys.argv like so:

if __name__ == '__main__':

    if len(sys.argv) == 5 and sys.argv[3] == "-c":
        exec(sys.argv[4])
    else:
        multiprocessing.set_start_method('forkserver')
        q = multiprocessing.Queue()
        TestProcess(q)
        time.sleep(5)
        print(q.get())

Then it show up in the process list as:

1000     11594  2.6  0.3  75836 10508 pts/2    S+   22:46   0:00 ./test
1000     11595  2.3  0.2  55892  7948 pts/2    S+   22:46   0:00 /home/danderson/testcx/dist/test -S -E -c from multiprocessing.semaphore_tracker import main;main(6)
1000     11596  2.6  0.3  75856 10280 pts/2    S+   22:46   0:00 /home/danderson/testcx/dist/test -S -E -c from multiprocessing.forkserver import main; main(6, 8, ['__main__'], **{'sys_path': ['/home/danderson/testcx/dist', '/home/danderson/testcx/dist/lib/python34.zip', '/home/danderson/testcx/dist/lib/python3.4/', '/home/danderson/testcx/dist/lib/python3.4/plat-linux', '/home/danderson/testcx/dist/lib/python3.4/lib-dynload']})

I am not parsing the -S or -E switches here, but it seems to work.

Another option may be to bundle the standard python interpreter somehow with cx_Freeze and set the path to it with multiprocessing.set_executable?

Has anyone come across this or know of any solutions, other than what I have above?

I am running python 3.4.3 and cx_Freeze 5.0.1 on Ubuntu.

Kind Regards,
David Anderson

[marcelotduarte]

You can test the patch in the latest development build:
pip install --upgrade --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze

I tested the PR in Ubuntu 20.04 Python 3.10.6 but I expect that it should work from Python 3.8 to 3.12.

[marcelotduarte]

Release 6.15.4 is out!