multiprocessing hangs when Pool used within Process #49581

zzzeek · 2009-02-20T16:06:51Z

BPO	5331

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-06-30.17:14:56.090>
created_at = <Date 2009-02-20.16:06:50.657>
labels = ['library', 'type-crash']
title = 'multiprocessing hangs when Pool used within Process'
updated_at = <Date 2009-06-30.17:14:56.088>
user = 'https://bugs.python.org/zzzeek'

bugs.python.org fields:

activity = <Date 2009-06-30.17:14:56.088>
actor = 'jnoller'
assignee = 'jnoller'
closed = True
closed_date = <Date 2009-06-30.17:14:56.090>
closer = 'jnoller'
components = ['Library (Lib)']
creation = <Date 2009-02-20.16:06:50.657>
creator = 'zzzeek'
dependencies = []
files = []
hgrepos = []
issue_num = 5331
keywords = []
message_count = 6.0
messages = ['82534', '82540', '89192', '89194', '89901', '89945']
nosy_count = 4.0
nosy_names = ['OG7', 'LambertDW', 'jnoller', 'zzzeek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue5331'
versions = ['Python 2.6', 'Python 3.0']

zzzeek · 2009-02-20T16:06:50Z

this occurs for me running on Mac OSX Leopard. The equivalent code
using "processing" in python 2.5 works fine, which is how I found this
bug - my code hung when upgraded to 2.6. Basically initiating a
multiprocessing.Pool inside of multiprocessing.Process hangs the
application. Below is code which illustrates the issue on both py2.6
and py3.0:

    from multiprocessing import Pool, Process
    import sys

    def f(x):
        return x*x

    def go():
        pool = Pool(processes=4)              
        x = pool.map(f, [1, 2, 3, 4, 5, 6, 7])
        sys.stdout.write("result: %s\n" % x)

    def go2():
        x = map(f, [1,2,3,4,5,6,7])
        sys.stdout.write("result: %s\n" % x)

    # pool alone, fine
    go()

    # process alone, fine
    p = Process(target=go2)
    p.start()
    p.join()

    # use both, hangs
    p = Process(target=go)
    p.start()
    p.join()

lambertdw · 2009-02-20T18:57:27Z

Fails also on this system.

$ p3
Python 3.0.1 (r301:69556, Feb 13 2009, 23:52:55) 
[GCC 3.4.6 20060404 (Red Hat 3.4.6-3)] on linux2
(...)
>>> # use both hangs
(...)
>>> p.start()
>>> p.join()
(hmm la  do diii   laaa...)^C
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/process.py",
line 119, in join
    res = self._popen.wait(timeout)
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 118, in wait
    return self.poll(0)
  File
"/usr/users/lambert/bin/python/lib/python3.0/multiprocessing/forking.py",
line 107, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
>>>

OG7 · 2009-06-10T08:58:09Z

Apparently the pool workers die all at once, just after the pool creates
them and before the pool is used.
I added a few lines to multiprocessing/pool.py to get the stack and the
exception backtrace.

        except (EOFError, IOError):
            import traceback
            debug(traceback.format_exc())
            debug(''.join(traceback.format_stack()))
            debug('worker got EOFError or IOError -- exiting')
            break

INFO::Rule dispatcher::multiprocessing::child process calling self.run()
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082559488
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3082104832
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081826304
DEBUG::Rule dispatcher::multiprocessing::created semlock with handle
3081822208
INFO::PoolWorker-3:1::multiprocessing::child process calling self.run()
DEBUG::PoolWorker-3:1::multiprocessing::Traceback (most recent call last):
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 57, in worker
    task = get()
  File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/queues.py",
line 339, in get
    return recv()
IOError: [Errno 9] Bad file descriptor

DEBUG::PoolWorker-3:1::multiprocessing:: File
"/home/who/var/co/git-svn/what-base/correlator/bin/correlator", line 30,
in <module>
what.corr.actors.main.main()
File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/main.py",
line 47, in main
rrp.start()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
self._popen = Popen(self)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
code = process_obj._bootstrap()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/home/who/var/co/git-svn/what-base/correlator/lib/what/corr/actors/rule_dispatcher.py",
line 26, in main
initargs=(manager.agg_msgs_queue, ))
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/init.py",
line 232, in Pool
return Pool(processes, initializer, initargs)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 107, in __init__
w.start()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 109, in start
self._popen = Popen(self)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/forking.py",
line 99, in __init__
code = process_obj._bootstrap()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 236, in _bootstrap
self.run()
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
File
"/home/who/.buildout/eggs/multiprocessing-2.6.1.1-py2.6-linux-i686.egg/multiprocessing/pool.py",
line 61, in worker
debug(''.join(traceback.format_stack()))

DEBUG::PoolWorker-3:1::multiprocessing::worker got EOFError or IOError
-- exiting
INFO::PoolWorker-3:1::multiprocessing::process shutting down
DEBUG::PoolWorker-3:1::multiprocessing::running all "atexit" finalizers
with priority >= 0

OG7 · 2009-06-10T10:42:32Z

It seems the root cause is at http://bugs.python.org/issue5155 .
A workaround is to use a duplex Pipe in SimpleQueue.

jnoller · 2009-06-30T03:04:15Z

Patch attached to bpo-5313, please review

jnoller · 2009-06-30T17:14:56Z

Committed in r73708 on trunk

zzzeek mannequin added stdlib Python modules in the Lib dir type-crash A hard crash of the interpreter, possibly with a core dump labels Feb 20, 2009

benjaminp assigned jnoller Feb 21, 2009

jnoller mannequin closed this as completed Jun 30, 2009

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multiprocessing hangs when Pool used within Process #49581

multiprocessing hangs when Pool used within Process #49581

zzzeek mannequin commented Feb 20, 2009

zzzeek mannequin commented Feb 20, 2009

lambertdw mannequin commented Feb 20, 2009

OG7 mannequin commented Jun 10, 2009

OG7 mannequin commented Jun 10, 2009

jnoller mannequin commented Jun 30, 2009

jnoller mannequin commented Jun 30, 2009

multiprocessing hangs when Pool used within Process #49581

multiprocessing hangs when Pool used within Process #49581

Comments

zzzeek mannequin commented Feb 20, 2009

zzzeek mannequin commented Feb 20, 2009

lambertdw mannequin commented Feb 20, 2009

OG7 mannequin commented Jun 10, 2009

OG7 mannequin commented Jun 10, 2009

jnoller mannequin commented Jun 30, 2009

jnoller mannequin commented Jun 30, 2009