multiprocessing fails to raise exception with parameters

[candlerb]

BPO	16558
Nosy	@bitdancer
Superseder	bpo-9400: multiprocessing.pool.AsyncResult.get() messes up exceptions

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-11-26.17:37:49.873>
created_at = <Date 2012-11-26.14:50:16.898>
labels = ['type-bug', 'library']
title = 'multiprocessing fails to raise exception with parameters'
updated_at = <Date 2012-11-26.17:37:49.872>
user = 'https://bugs.python.org/candlerb'

bugs.python.org fields:

activity = <Date 2012-11-26.17:37:49.872>
actor = 'sbt'
assignee = 'none'
closed = True
closed_date = <Date 2012-11-26.17:37:49.873>
closer = 'sbt'
components = ['Library (Lib)']
creation = <Date 2012-11-26.14:50:16.898>
creator = 'candlerb'
dependencies = []
files = []
hgrepos = []
issue_num = 16558
keywords = []
message_count = 3.0
messages = ['176416', '176422', '176433']
nosy_count = 3.0
nosy_names = ['r.david.murray', 'sbt', 'candlerb']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '9400'
type = 'behavior'
url = 'https://bugs.python.org/issue16558'
versions = ['Python 2.7']

[candlerb]

Probably best demonstrated by example.

~~~~~~~~~~~~~~~~
import multiprocessing

class Myerror(ValueError):
    def __init__(self,a):
        self.a = a
    def __str__(self):
        return repr(self.a)
        
def foo(arg):
    raise Myerror(arg)
    
#foo("1")   #<= this works fine, raises exception as expected

#But this breaks:
pool = multiprocessing.Pool(2)
pool.map(foo, ["1","2","3"])
~~~~~~~~~~~~~~~~

The result seen:

~~~~~~~~~~~~~~~~
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 353, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.Myerror'>, ())
~~~~~~~~~~~~~~~~

At this point the application hangs. Worse: pressing ctrl-C shows a traceback and KeyboardInterrupt, but the worker keeps getting restarted, so it's impossible to stop. You have to go to another shell and do somthing like

killall python

to terminate the program.

A real-world example (which caused me to track this down) is a CalledProcessError raised by subprocess.check_call

~~~~~~~~~~~~~~~~
import multiprocessing
import subprocess

def foo(arg):
    subprocess.check_call("nonexistent", shell=True)
    #raise subprocess.CalledProcessError(127, "nonexistent")

pool = multiprocessing.Pool(2)
pool.map(foo, ["1","2","3"])
~~~~~~~~~~~~~~~~

which fails in the same way:

~~~~~~~~~~~~~~~~
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 353, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
~~~~~~~~~~~~~~~~

Behaviour tested on:
python 2.7.3 on Ubuntu 12.04
python 2.7.1 on OSX 10.7.5

Workaround: re-raise a parameter-less exception instead.

try:
    ...
except Exception as e:
    raise RuntimeError

[bitdancer]

This is probably related to bpo-1692335. It looks like that fix was not backported. Can you test if your example works now in 3.3?

[sbt]

The example works correctly on 3.3 because of bpo-1692335. I am not sure if it is appropriate to backport it though.

This is a duplicate of bpo-9400 which I have assigned to myself. (I had thought it was already fixed.)