multiprocessing exceptions with useful traceback

[andresriancho]

BPO	17836
Files	mp_exception_bug.py: Reproduce multiprocessing exception traceback bug

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-04-25.21:23:00.075>
created_at = <Date 2013-04-24.23:32:01.803>
labels = ['library']
title = 'multiprocessing exceptions with useful traceback'
updated_at = <Date 2013-04-25.21:23:00.074>
user = 'https://bugs.python.org/AndresRiancho'

bugs.python.org fields:

activity = <Date 2013-04-25.21:23:00.074>
actor = 'sbt'
assignee = 'none'
closed = True
closed_date = <Date 2013-04-25.21:23:00.075>
closer = 'sbt'
components = ['Library (Lib)']
creation = <Date 2013-04-24.23:32:01.803>
creator = 'Andres.Riancho'
dependencies = []
files = ['30009']
hgrepos = []
issue_num = 17836
keywords = []
message_count = 3.0
messages = ['187751', '187752', '187817']
nosy_count = 2.0
nosy_names = ['Andres.Riancho', 'sbt']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue17836'
versions = ['Python 3.4']

[andresriancho]

In pool.py, the worker function reads as follows:

http://svn.python.org/view/python/trunk/Lib/multiprocessing/pool.py?view=markup
"""
68 job, i, func, args, kwds = task
69 try:
70 result = (True, func(*args, **kwds))
71 except Exception, e:
72 result = (False, e)
...
488 if self._success:
489 return self._value
490 else:
491 raise self._value
"""

If an exception is raised in the function you sent to the pool, the exception you get has "raise self._value" as the last line; which is correct but useless for debugging.

mp_exception_bug.py reproduces this error. This is the output:

"""
Traceback (most recent call last):
  File "mp_exception_bug.py", line 8, in <module>
    print p.map(f, [1,2,3])
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
    raise self._value
NameError: global name 'y' is not defined
"""

As you can imagine, "NameError: global name 'y' is not defined" is not enough in complex projects.

If we apply some changes to the pool.py we could get something similar to this:

"""
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 98, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 67, in mapstar
    return map(*args)
  File "mp_exception_bug.py", line 4, in f
    return x*y
NameError: global name 'y' is not defined

Traceback (most recent call last):
  File "mp_exception_bug.py", line 8, in <module>
    print p.map(f, [1,2,3])
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 231, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 535, in get
    raise self._value[0]
NameError: global name 'y' is not defined
"""

The patch is simple but ugly:

"""

import sys
import traceback
72c100,102
< result = (False, e)
---
exc_info = sys.exc_info()
tb_string = traceback.format_exc(exc_info[2])
result = (False, (e, tb_string))
491c532,535
< raise self._value
---
# Do something with the exception here, the simplest (but ugliest)
# thing to do is to simply print it to the console
print self._value[1]
raise self._value[0]
"""

Note that I tried to replace the "raise self._value[0]" with a raise with three parameters, being the last one the traceback we get using "exc_info = sys.exc_info()" but sadly it is not possible to pickle tracebacks.

I understand that printing is not the best thing to do here, but I wanted to get this discussion started and come to a real solution.

Thanks

[ned-deily]

(FYI, cpython source is no longer maintained in svn.python.org; the browser link you provided is out-of-date. Current source for the 2.7 branch can be viewed here: http://hg.python.org/cpython/file/2.7/)

[sbt]

Duplicate of bpo-13831.