Multiprocessing infinite loop

[Kain94]

BPO	8094
Nosy	@amauryfa, @bitdancer, @applio
Files	multiprocessing.rst.diff: Patch to the multiprocessing documentation file

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-29.21:21:04.715>
created_at = <Date 2010-03-09.09:01:52.629>
labels = ['type-bug', 'docs']
title = 'Multiprocessing infinite loop'
updated_at = <Date 2015-02-10.14:49:53.365>
user = 'https://bugs.python.org/Kain94'

bugs.python.org fields:

activity = <Date 2015-02-10.14:49:53.365>
actor = 'davin'
assignee = 'jnoller'
closed = True
closed_date = <Date 2015-01-29.21:21:04.715>
closer = 'davin'
components = ['Documentation']
creation = <Date 2010-03-09.09:01:52.629>
creator = 'Kain94'
dependencies = []
files = ['21603']
hgrepos = []
issue_num = 8094
keywords = ['patch']
message_count = 10.0
messages = ['100707', '100724', '100731', '100733', '133468', '134187', '134190', '181088', '181089', '234991']
nosy_count = 9.0
nosy_names = ['amaury.forgeotdarc', 'jnoller', 'r.david.murray', 'asksol', 'Kain94', 'dsdale24', 'Rodrigue.Alcazar', 'pthiem', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'works for me'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8094'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[Kain94]

Hi,

The following code results in an infinite loop -->

# ----

import multiprocessing

def f(m):
	print(m)

p = multiprocessing.Process(target=f, args=('pouet',))
p.start()
# 

---

I've firstly think about an issue in my code when Python loads this module so I've added a "if __name__ == '__main__':" and it works well. But, with the following code Python enters in the same infinite loop -->

proc.py:

# ----

import multiprocessing

def f(m):
	print(m)

class P:
	def __init__(self, msg):
		self.msg = msg
	
	def start(self):
		p = multiprocessing.Process(target=f, args=(self.msg,))
		p.start()
# 

---

my_script.py:
# ----

from proc import P

p = P("pouet")
p.start()
# 

---

It's like Python loads all parent's process memory space before starting process which issues in an infinite call to Process.start() ...

[bitdancer]

Are you running this on windows?

[Kain94]

Sorry I've not made clear my working platform.

Yes, I'm running Python 3.1.1 32 bit on a Windows 7 x64.

[amauryfa]

The restriction that imposes the "__name__= '__main__'" idiom also applies when multiprocessing is not used in the main module.

Actually the main module is always reloaded in the subprocess. The docs should be more explicit about it.

[RodrigueAlcazar]

I have tried to clearly state that the main module is imported by a newly created process. I have also added a comment explaining that an infinite loop like the one Benjamin describes could be created.

[dsdale24]

I think I have a similar situation:

C:\Py\Scripts\foo
---

if __name__ == '__main__':
    import bar
    bar.main()

C:\Py\Lib\site-packages\bar.py
---

from multiprocessing import Pool

def task(arg):
    return arg

def main():
    pool = Pool()
    res = pool.apply_async(task, (3.14,))

print res.get()

if __name__ == '__main__':
    main()

I can run "python bar.py". "python C:\Py\Scripts\foo" yields an infinite stream of errors:

File "<string>", line 1 in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 346, in main
prepare(preparation_data)
File "C:\Python27\lib\multiprocessing\forking.py", line 455, in prepare
file, path_name, etc = imp.find_module(main_name, dirs)
ImportError: No module named foo

This same scheme works fine on linux. Have I just overlooked something simple?

[bitdancer]

Your code doesn't appear to have anything to do with the reported bug, which is about an infinite loop. But FYI to my understanding your script can't work on windows, since foo can't be imported. On linux, foo doesn't need to be imported.

[pthiem]

As an alternative, see http://bugs.python.org/issue10845

It contains a patch for the 3.X series which fixes the infinity loop.

Applying it to the 2.X series will fix the issue and make a change the documentation unnecessary.

[pthiem]

Actually sorry, now that I reread the details a second time, I'm not sure that is this the same deal. I'll just file a separate bug.

[applio]

On Windows 7 (64-bit), it was not possible to reproduce any infinite looping behavior with the supplied example code.

Specifically, with the two examples from Benjamin, the observed behavior when running them was the same under 2.7.8 and default (3.5): a RuntimeError results with a message suggesting that the code perhaps needs to leverage the "if __name__ == '__main__'" idiom to avoid having both parent and all subsequent children processes starting up a new process because they're all unintentionally running the same lines of code intended only for the parent to run. Adding that idiomatic test to each of the two examples permits them to run to a happy conclusion. That is, in the case of the first example we make that one-line code change to read:
# ---

import multiprocessing

def f(m):
    print(m)

if __name__ == '__main__':
    p = multiprocessing.Process(target=f, args=('pouet',))
    p.start()
# 

---

This would be a recommended practice on unix-y systems as well as Windows.

Aside: It was not possible to reproduce the issue injected by Darren either -- perhaps the example code provided was not quite what he intended.

The infinite looping behavior described in the original issue description might well have been reproducible in much earlier releases. In the current default (3.5) branch (or in 2.7.8), it is no longer possible to reproduce. I'm tempted to mark this as "out of date" but instead will opt for "works for me" and close the issue.