Recipe Multiprocessing

Martin Zibricky edited this page Sep 25, 2015 · 5 revisions

Required Changes to your code

When using the multiprocessing module, you must call

multiprocessing.freeze_support()

straight after the if __name__ == '__main__' line of the main module. This change is sufficient for the default (--onedir) mode.

Please read the Python library manual about multiprocessing.freeze_support for more information.

Multiprocessing code fails when using a --onefile executable on Windows

NB: This problem is specific to Windows, which does not support spawn(). It does not occur on other (Posix) platforms like all flavors of Unix and Mac OS X.

For using python module multiprocess on Windows, you need to extend your multiprocessing code as shown below. See this thread about the background and ticket https://github.com/pyinstaller/pyinstaller/issues/182 for more information.

This recipe requires PyInstaller 3.0 or later.

import os
import sys

# Module multiprocessing is organized differently in Python 3.4+
try:
    # Python 3.4+
    if sys.platform.startswith('win'):
        import multiprocessing.popen_spawn_win32 as forking
    else:
        import multiprocessing.popen_fork as forking
except ImportError:
    import multiprocessing.forking as forking

if sys.platform.startswith('win'):
    # First define a modified version of Popen.
    class _Popen(forking.Popen):
        def __init__(self, *args, **kw):
            if hasattr(sys, 'frozen'):
                # We have to set original _MEIPASS2 value from sys._MEIPASS
                # to get --onefile mode working.
                os.putenv('_MEIPASS2', sys._MEIPASS)
            try:
                super(_Popen, self).__init__(*args, **kw)
            finally:
                if hasattr(sys, 'frozen'):
                    # On some platforms (e.g. AIX) 'os.unsetenv()' is not
                    # available. In those cases we cannot delete the variable
                    # but only set it to the empty string. The bootloader
                    # can handle this case.
                    if hasattr(os, 'unsetenv'):
                        os.unsetenv('_MEIPASS2')
                    else:
                        os.putenv('_MEIPASS2', '')

    # Second override 'Popen' class with our modified version.
    forking.Popen = _Popen


# Example for testing multiprocessing.

import multiprocessing

class SendeventProcess(multiprocessing.Process):
    def __init__(self, resultQueue):
        self.resultQueue = resultQueue
        multiprocessing.Process.__init__(self)
        self.start()

    def run(self):
        print 'SendeventProcess'
        self.resultQueue.put((1, 2))
        print 'SendeventProcess'


if __name__ == '__main__':
    # On Windows calling this function is necessary.
    # On Linux/OSX it does nothing.
    multiprocessing.freeze_support()
    print 'main'
    resultQueue = multiprocessing.Queue()
    SendeventProcess(resultQueue)
    print 'main'

Console output of this code snippet should be similar to

main
main
SendeventProcess
SendeventProcess