Multiprocessing Manager Class Causes Infinite Loop on Mac OS (intel) #2435
Comments
|
After looking into the source code of the hook function for the multiprocessing module, I read the comment about is_darwin = True # This is determined by a module that does not affect the results.
def main():
...
if __name__ == "__main__":
if is_darwin:
import multiprocessing.spawn
import multiprocessing
multiprocessing.spawn.freeze_support() # This fixes the issue.
print("Creating Managers")
manager_1 = multiprocessing.Manager()
print("Finished Creating Manager #1")
manager_2 = multiprocessing.Manager()
print("Finished Creating Manager #2")
manager_3 = multiprocessing.Manager()
print("Finished Creating Manager #3")
print("Finished Creating ALL Managers.")
main() |
|
I use 3 tests on Windows, Linux, and macOS. |
|
I reviewed all of your test cases with the following setup. [sample1.py] import multiprocessing, sys
def foo(q):
q.put('hello')
if __name__ == '__main__':
if sys.platform == 'win32': # the conditional is unecessary
multiprocessing.freeze_support()
multiprocessing.set_start_method('spawn')
q = multiprocessing.SimpleQueue()
p = multiprocessing.Process(target=foo, args=(q,))
p.start()
print(q.get())
p.join()[sample2.py] import multiprocessing, sys
def foo(q):
q.put('hello')
if __name__ == '__main__':
ctx = multiprocessing.get_context('spawn')
if sys.platform == 'win32': # the conditional is unecessary
ctx.freeze_support()
q = ctx.Queue()
p = ctx.Process(target=foo, args=(q,))
p.start()
print(q.get())
p.join()[sample3.py] if __name__ == "__main__":
import multiprocessing, sys
if sys.platform == 'win32': # the conditional is unecessary
multiprocessing.freeze_support()
multiprocessing.set_start_method('spawn')
mgr = multiprocessing.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")[setup.py] from cx_Freeze import Executable, setup
setup(
name="test_multiprocessing",
version="0.1",
description="Sample for test with cx_Freeze",
executables=[
Executable("sample1.py"),
Executable("sample2.py"),
Executable("sample3.py"),
],
options={
"build_exe": {
"excludes": ["tkinter"],
"silent": True,
}
}
)[sample1.py Run Command] ./build/exe.macos-10.9-universal2-3.12/sample1
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample1[sample2.py Run Command] ./build/exe.macos-10.9-universal2-3.12/sample2
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample2[sample3.py Run Command] ./build/exe.macos-10.9-universal2-3.12/sample3
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3[setup.py Run Command] python setup.py bdist_macOutput Alternate Test Case For Sample 3 if __name__ == "__main__":
import multiprocessing, sys
if sys.platform == 'win32': # the conditional is unecessary
multiprocessing.freeze_support()
# multiprocessing.set_start_method('spawn') Removed for testing purposes
print("Creating Manager")
mgr = multiprocessing.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")[Run Command] ./build/exe.macos-10.9-universal2-3.12/sample3
./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3[Output] user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample3
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...Conclusion |
|
Your conclusion is interesting, because according to the Python documentation, spawn is the default start method. Then there would be no need to configure it, and in the set_start_method documentation it states "that this should be called at most once". Also, in the The spawn and forkserver start methods section has "Safe importing of main module" with recommendations to use set_start_method. Can you check if the default start method is indeed spawn in python and in the executable? |
|
|
|
[sample4.py] if __name__ == "__main__":
import multiprocessing
mp_context = multiprocessing.get_context('spawn')
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")Output user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample3
Creating Manager
creating dict...done!user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample3
Creating Manager
creating dict...done!Alternate Test Case For Sample 4 if __name__ == "__main__":
import multiprocessing
#mp_context = multiprocessing.get_context('spawn') Removed for test purposes
print("Creating Manager")
mgr = multiprocessing.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")Output of Alternate Test Case user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
Creating Manager
Creating Manager
Creating Manager
Creating Manager
Creating Manager
...The Default Start Method if __name__ == "__main__":
import multiprocessing
print(multiprocessing.get_start_method()) # The default start method
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")Output user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
spawn
Creating Manager
spawn
Creating Manager
spawn
Creating Manager
...user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
spawn
Creating Manager
spawn
Creating Manager
spawn
Creating Manager
...Another Test Case with Getting the Default Spawn Method if __name__ == "__main__":
import multiprocessing.spawn
import multiprocessing
multiprocessing.spawn.freeze_support()
print(multiprocessing.get_start_method()) # The default start method
print("Creating Manager")
mgr = mp_context.Manager()
var = [1] * 10000000
print("creating dict", end="...")
mgr_dict = mgr.dict({'test': var})
print("done!")Output user@user-ImMac multiprocessing % ./build/exe.macos-10.9-universal2-3.12/sample4
spawn
Creating Manager
creating dict...done!user@user-iMac multiprocessing % ./build/test_multiprocessing-0.1.app/Contents/MacOS/sample4
spawn
Creating Manager
creating dict...done!Off Topic Question |
|
Trimmed example for context: from multiprocessing import Process, freeze_support
def f():
print("Hello from cx_Freeze")
if __name__ == "__main__":
freeze_support()
Process(target=f).start() |
|
The default in python is |
|
@IsaiahCoroama read this. @ntindle I should change the FAQ or redirect freeze_support in the hook to avoid future questions.... |
|
My naive understanding is that you sometimes need one vs the other depending on if you want to spawn or fork which differs by platform |
Apologies for the confusion. You're right, it should be |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Uninstall cx_Freeze and reinstall to check the wheel installed, or download https://files.pythonhosted.org/packages/e8/12/70285b18f9dea354e69ee1b472f46e79f9ae9aa6749d23a3109bb73aa9fc/cx_Freeze-7.1.0.post0-cp312-cp312-macosx_10_9_universal2.whl and install it, so you should get a universal2 executable. |
|
Based on information from you and others, I improved the hook for multiprocessing. |
|
Release 7.1.1 is out! |
Bug Description
When I try to create a
multiprocessing.Manager()object, even withfreeze_support, it causes an infinite loop.Steps To Reproduce
Create a
main.pyfile.Create a
setup.pyfile.Run the setup file with the following configuration:
Run the generated application file:
Expected behavior
From the demo, it prints "Creating Managers" repeatedly indefinity.
System Information:
Mac OS (Sonoma 14.4)x86_647.1.0.post03.12.3Additional context
Within the project, I am using managers to share data between three separate processes. These processes run a loop attached to a
manager.Event()object. In the loop, it checks if a boolean value within amanager.dict()object is True. If it is, it will call atkinterfunction; these consist offiledialog.askdirectory(),messagebox.showerror(), andsimpledialog.askstring(). Then, it sets the output to a value within the manager dictionary. The process is then repeated with a few other steps. If you would like, I can post the full source code, but currently, the issue only remains with creating the first manager object. I have gotten this to work withPyInstaller, but I need to create auniversal2application, andcx_Freezehas been the closest I have gotten to finally completing everything I need.Although it states in the official documentation that
multiprocessing.freeze_support()only works on Windows, it has fixed this same infinite loop withinPyInstallerwhen I was trying to get that to work.Disclaimer
If there are any misspellings within the code above, that is an accident and not in my actual code. I wrote this issue from my Windows machine all by hand.
The text was updated successfully, but these errors were encountered: