Multiprocessing hangs when multiprocessing.Pool methods are called

[mishok13]

BPO	3272

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-01-22.19:25:51.185>
created_at = <Date 2008-07-03.16:48:37.613>
labels = ['library', 'type-crash']
title = 'Multiprocessing hangs when multiprocessing.Pool methods are called'
updated_at = <Date 2009-01-22.19:25:51.184>
user = 'https://bugs.python.org/mishok13'

bugs.python.org fields:

activity = <Date 2009-01-22.19:25:51.184>
actor = 'jnoller'
assignee = 'jnoller'
closed = True
closed_date = <Date 2009-01-22.19:25:51.185>
closer = 'jnoller'
components = ['Library (Lib)']
creation = <Date 2008-07-03.16:48:37.613>
creator = 'mishok13'
dependencies = []
files = []
hgrepos = []
issue_num = 3272
keywords = []
message_count = 2.0
messages = ['69207', '80375']
nosy_count = 3.0
nosy_names = ['roudkerk', 'jnoller', 'mishok13']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue3272'
versions = ['Python 2.6', 'Python 3.0']

[mishok13]

multiprocessing hangs, when multiprocessing.Pool methods map,
imap, imap_unordered, apply, apply_async, map_async are called.
Here is an example:

Python 3.0b1+ (py3k:64686M, Jul  3 2008, 13:06:13)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing import Pool
>>> Pool(processes=4).imap(lambda x: x, range(10))
<multiprocessing.pool.IMapIterator object at 0x83bcc8c>
>>> Pool(processes=4).map(lambda x: x, range(10))
Exception in thread Thread-13:
Traceback (most recent call last):
  File "/usr/local/lib/python3.0/threading.py", line 492, in
_bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.0/threading.py", line 447, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.0/multiprocessing/pool.py", line 225, in
_handle_tasks
    put(task)
  File "/usr/local/lib/python3.0/pickle.py", line 1319, in dumps
    Pickler(f, protocol).dump(obj)
  File "/usr/local/lib/python3.0/multiprocessing/util.py", line 311, in
_reduce_method
    if m.__self__ is None:
AttributeError: 'function' object has no attribute '__self__'

Process PoolWorker-28:
Traceback (most recent call last):
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 232,
in _bootstrap
    self.run()
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 88,
in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.0/multiprocessing/pool.py", line 57, in
worker
    task = get()
  File "/usr/local/lib/python3.0/multiprocessing/queues.py", line 337,
in get
Process PoolWorker-27:
Traceback (most recent call last):
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 232,
in _bootstrap
    self.run()
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 88,
in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.0/multiprocessing/pool.py", line 57, in
worker
    task = get()
  File "/usr/local/lib/python3.0/multiprocessing/queues.py", line 337,
in get
Process PoolWorker-26:
Traceback (most recent call last):
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 232,
in _bootstrap
Process PoolWorker-25:
Traceback (most recent call last):
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 232,
in _bootstrap
    self.run()
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 88,
in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.0/multiprocessing/pool.py", line 57, in
worker
    task = get()
  File "/usr/local/lib/python3.0/multiprocessing/queues.py", line 337,
in get
    self.run()
  File "/usr/local/lib/python3.0/multiprocessing/process.py", line 88,
in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.0/multiprocessing/pool.py", line 57, in
worker
    task = get()
  File "/usr/local/lib/python3.0/multiprocessing/queues.py", line 339,
in get
    return recv()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
    racquire()
KeyboardInterrupt
Terminated

The same happens on latest trunk of Python 2.6.
Note, map and apply methods hang Python interpreter completely, so I
have to use kill utility to exit Python.

[jnoller]

This is now covered in the multiprocessing documentation:

"Note Functionality within this package requires that the __main__ method
be importable by the children. This is covered in Programming guidelines
however it is worth pointing out here. This means that some examples, such
as the multiprocessing.Pool examples will not work in the interactive
interpreter."