Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

truncate potentially long CompositeErrors #2871

Merged
merged 2 commits into from

5 participants

@minrk
Owner

If you have a typo in a task on 100 engines, the local exception will be a bit ridiculous.

This defaults the truncation to 4 tracebacks (could be even fewer, or even one?), and you can set the tb_limit attribute of CompositeError to adjust this.

Prompted by this SO question.

  • document / add message on how to change the default traceback number
@minrk minrk truncate potentially long CompositeErrors
If you have a typo in a task on 100 engines, the local exception will be a bit ridiculous.

This defaults the truncation to 4 tracebacks (could be even fewer, or even one?),
and you can set the tb_limit attribute of CompositeError to adjust this.

Prompted by [this SO question](http://stackoverflow.com/questions/14635935/silence-or-condence-ipython-parallel-exceptions).
529df40
@parente

I asked on SO because I figured the knob was there and I was foolishly missing it. Thanks for adding it!

@Carreau
Owner

Maybe drop a line in the doc for tb_limit or add
' set CompositeError.tb_limit to change the number of shown traceback'
after
'... %i more exceptions ...'

Thought ?

@minrk
Owner

Probably a good idea, thanks.

@minrk minrk was assigned
@ellisonbg
Owner

This code looks good and the Travis build passes. Once you add the docs, I would go ahead with the merge.

@minrk
Owner

doc pushed

@ellisonbg ellisonbg merged commit c2bc10d into from
@minrk minrk deleted the branch
@minrk
Owner

gah, sorry, I'll fix that shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 1, 2013
  1. @minrk

    truncate potentially long CompositeErrors

    minrk authored
    If you have a typo in a task on 100 engines, the local exception will be a bit ridiculous.
    
    This defaults the truncation to 4 tracebacks (could be even fewer, or even one?),
    and you can set the tb_limit attribute of CompositeError to adjust this.
    
    Prompted by [this SO question](http://stackoverflow.com/questions/14635935/silence-or-condence-ipython-parallel-exceptions).
Commits on Feb 20, 2013
  1. @minrk
This page is out of date. Refresh to see the latest.
View
14 IPython/parallel/error.py
@@ -232,6 +232,8 @@ class TaskRejectError(KernelError):
class CompositeError(RemoteError):
"""Error for representing possibly multiple errors on engines"""
+ tb_limit = 4 # limit on how many tracebacks to draw
+
def __init__(self, message, elist):
Exception.__init__(self, *(message, elist))
# Don't use pack_exception because it will conflict with the .message
@@ -256,22 +258,28 @@ def _get_traceback(self, ev):
def __str__(self):
s = str(self.msg)
- for en, ev, etb, ei in self.elist:
+ for en, ev, etb, ei in self.elist[:self.tb_limit]:
engine_str = self._get_engine_str(ei)
s = s + '\n' + engine_str + en + ': ' + str(ev)
+ if len(self.elist) > self.tb_limit:
+ s = s + '\n.... %i more exceptions ...' % (len(self.elist) - self.tb_limit)
return s
def __repr__(self):
- return "CompositeError(%i)"%len(self.elist)
+ return "CompositeError(%i)" % len(self.elist)
def render_traceback(self, excid=None):
"""render one or all of my tracebacks to a list of lines"""
lines = []
if excid is None:
- for (en,ev,etb,ei) in self.elist:
+ for (en,ev,etb,ei) in self.elist[:self.tb_limit]:
lines.append(self._get_engine_str(ei))
lines.extend((etb or 'No traceback available').splitlines())
lines.append('')
+ if len(self.elist) > self.tb_limit:
+ lines.append(
+ '... %i more exceptions ...' % (len(self.elist) - self.tb_limit)
+ )
else:
try:
en,ev,etb,ei = self.elist[excid]
View
27 IPython/parallel/tests/test_view.py
@@ -619,6 +619,33 @@ def test_compositeerror_render_exception(self):
self.assertEqual(io.stdout.count('by zero'), len(view), io.stdout)
self.assertEqual(io.stdout.count(':execute'), len(view), io.stdout)
+ def test_compositeerror_truncate(self):
+ """Truncate CompositeErrors with many exceptions"""
+ view = self.client[:]
+ msg_ids = []
+ for i in range(10):
+ ar = view.execute("1/0")
+ msg_ids.extend(ar.msg_ids)
+
+ ar = self.client.get_result(msg_ids)
+ try:
+ ar.get()
+ except error.CompositeError as e:
+ pass
+ else:
+ self.fail("Should have raised CompositeError")
+
+ lines = e.render_traceback()
+ with capture_output() as io:
+ e.print_traceback()
+
+ self.assertTrue("more exceptions" in lines[-1])
+ count = e.tb_limit
+
+ self.assertEqual(io.stdout.count('ZeroDivisionError'), 2 * count, io.stdout)
+ self.assertEqual(io.stdout.count('by zero'), count, io.stdout)
+ self.assertEqual(io.stdout.count(':execute'), count, io.stdout)
+
@dec.skipif_not_matplotlib
def test_magic_pylab(self):
"""%pylab works on engines"""
View
63 docs/source/parallel/parallel_multiengine.txt
@@ -560,25 +560,25 @@ more other types of exceptions. Here is how it works:
In [79]: dview.execute("1/0")
[0:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
[1:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
[2:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
[3:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
@@ -595,7 +595,7 @@ If you want, you can even raise one of these original exceptions:
....:
....:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
@@ -608,25 +608,25 @@ instance:
In [81]: dview.execute('1/0')
[0:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
[1:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
[2:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
[3:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
@@ -654,11 +654,31 @@ instance:
ipdb> e.print_traceback(1)
[1:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
+Since you might have 100 engines, you probably don't want to see 100 tracebacks
+for a simple NameError because of a typo.
+For this reason, CompositeError truncates the list of exceptions it will print
+to :attr:`CompositeError.tb_limit` (default is five).
+You can change this limit to suit your needs with:
+
+.. sourcecode:: ipython
+
+ In [20]: from IPython.parallel import CompositeError
+ In [21]: CompositeError.tb_limit = 1
+ In [22]: %px a=b
+ [0:execute]:
+ ---------------------------------------------------------------------------
+ NameError Traceback (most recent call last)
+ ----> 1 a=b
+ NameError: name 'b' is not defined
+
+ ... 3 more exceptions ...
+
+
All of this same error handling magic even works in non-blocking mode:
.. sourcecode:: ipython
@@ -670,25 +690,8 @@ All of this same error handling magic even works in non-blocking mode:
In [85]: ar.get()
[0:execute]:
---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
- ----> 1 1/0
- ZeroDivisionError: integer division or modulo by zero
-
- [1:execute]:
- ---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
+ ZeroDivisionError Traceback (most recent call last)
----> 1 1/0
ZeroDivisionError: integer division or modulo by zero
-
- [2:execute]:
- ---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
- ----> 1 1/0
- ZeroDivisionError: integer division or modulo by zero
-
- [3:execute]:
- ---------------------------------------------------------------------------
- ZeroDivisionError Traceback (most recent call last)<ipython-input-1-05c9758a9c21> in <module>()
- ----> 1 1/0
- ZeroDivisionError: integer division or modulo by zero
-
+
+ ... 3 more exceptions ...
Something went wrong with that request. Please try again.