Skip to content
This repository

protect IPython from bad custom exception handlers #876

Merged
merged 7 commits into from over 2 years ago

2 participants

Min RK Fernando Perez
Min RK
Owner

Previously, errors in custom handlers would result in the custom exception
handler's error being printed in lieu of the real exception, and certain cases could cause infinite loops.

Now, if CustomTB fails it is unregistered immediately, and the original TB is also displayed.

IPython's own BdbQuit_IPython_excepthook had an invalid signature, which revealed this issue, and has also been fixed.

test included.

closes #692

added some commits October 14, 2011
Min RK protect IPython from bad custom exception handlers
Previously, errors in custom handlers would result in the custom exception
handler's error being printed in lieu of the real exception, and certain cases could cause infinite loops.

Now, if CustomTB fails it is unregistered immediately, and the original TB is also displayed.

IPython's own BdbQuit_IPython_excepthook had an invalid signature, which revealed this issue, and has also been fixed.

test included.

closes #692
ed5078c
Min RK show tb before entering debug when %pdb is on
closes #690
0d065b1
Min RK
Owner

While looking at the same code, I saw the culprit for #690 as well, so I fixed that, too.

Min RK prevent atexit handlers from generating crash report
register `sys.excepthook = sys.__excepthook__` with atexit on aplication startup,
so it should be the first handler called.  This removes the crash handler.

closes #207
d5548fa
Fernando Perez
Owner

Code looks good, but this simple snippet

from IPython.core.debugger import Tracer
Tracer()

currently exits silently. I've never actually used Tracer() myself, but I'd expect it to activate the debugger in tracing mode, yet it doesn't. Based on the discussion of #692, does this really close the ticket?

Min RK protect against bad return type of CustomTB
includes test
also expands a few docstrings
09803ad
Min RK
Owner

Tracer() is a callable object that can be invoked to set_trace.

I did find a typo to fix, and added return-type validation, which I just pushed.

This is how Tracer appears to be used, based on the source:

from IPython.core.debugger import Tracer
tracer = Tracer()

try:
    1/0
except:
    tracer()
Fernando Perez
Owner

Question, shouln't we be trapping the quit exception? Trying out the example you show, I'm getting this when I quit:

dreamweaver[~]> python mini.py 
--Return--
None
> /home/fperez/mini.py(7)()
      6 except:
----> 7     tracer()
      8 

ipdb> q
Traceback (most recent call last):
  File "mini.py", line 7, in 
    tracer()
  File "/usr/lib/python2.6/bdb.py", line 50, in trace_dispatch
    return self.dispatch_return(frame, arg)
  File "/usr/lib/python2.6/bdb.py", line 84, in dispatch_return
    if self.quitting: raise BdbQuit
bdb.BdbQuit

I have vague memories of, long ago, having written code in ipdb to handle that case, but maybe it's not in the right place when used in standalone tracing mode...

Min RK
Owner

It does handle BdbQuit correctly from IPython. I don't know why it doesn't in regular Python. The exception handler is set when outside IPython here.

Adding a print statement to the excepthook, it is never called.

Min RK
Owner

Nevermind, I found it - ipapi.get() is not an acceptable way to check if you are in IPython - it will always return an IPython shell. get_ipython is the way to check if you are currently in IPython.

Min RK
Owner

Should I also do something about #636, while I'm cleaning up debugger miscellany?

pydb checks for '-pydb' in sys.argv, but there is no pydb flag anymore. Should I add --pydb as a flag to Shell Apps?

Min RK
Owner

Looking into pydb, it would appear that the project is entirely deprecated (files removed from pypi, even), in favor of a rewrite called pydbgr, which is not even installable as far as I can tell, and also appears essentially abandoned.

Min RK re-enable pydb flag
Note that pydb has been deprecated, and superseded by pydbgr, which may be
abandoned.

It would be preferable if the Pdb class could inherit from pydb or pdb
based on a runtime flag rather than checking sys.argv at the top level.
This at least restores old behavior for pydb users.

closes #636
d16b38e
Min RK
Owner

I re-enabled the pydb flag, so it should work. It doesn't set any config, it just allows the flag to pass through, so debugger.py can make its check. Ultimately this should probably be fixed so the check doesn't need to be run at import-time, but for now, at least, we don't have a never-true invalid flag check.

Fernando Perez
Owner

This is looking pretty good. Only one note: 09c2492 makes the correct check, but by not instantiating a full ipython, now the ipdb instance comes up without coloring. It would be nice to restore that, though without waiting to fire up a full blown ipython. That would mean parsing the config for colors...

If you think you can take a quick shot at reading the color settings correctly to instantiate the tracer with color active and matching the user's choice, that would be great.

But if it's looking like too much work, don't sweat it. I don't want to hold this PR forever on that little bit of nicety, so I'm happy to just leave an issue open for color support in the tracer and move on as-is. Your call.

Min RK
Owner
Min RK
Owner

It's not quite as simple as I thought.

There are two defaults:

  • load_default_config() will load your default profile
  • InteractiveShell.colors.get_default_value() will get the default value from the class, sans config.

It's easy to do the latter, but it turns out the former is less easy (or at least less clean), due to Class heirarchy, as we really use TerminalInteractiveShell, a subclass of InteractiveShell, both having a colors argument (again, colors must eventually be removed from the InteractiveShell object, as they are a purely frontend notion, but that's not relevant here).

So, to match the default resolution of the TerminalInteractiveShell's colors with default config would be:

            cfg = load_default_config()
            try:
                def_colors = cfg.TerminalInteractiveShell.colors
            except AttributeError:
                try:
                    def_colors = cfg.InteractiveShell.colors
                except AttributeError:
                    def_colors = InteractiveShell.colors.get_default_value()

And any changes to inheritance could muck this up.

Min RK
Owner

In the debugger code, the default colors are hardcoded to NoColor (Tracer has a colors parameter, while Pdb has color_scheme), is it possible that's deliberate and/or desirable? I can easily change them to use InteractiveShell's default. I think using config files should be put off until the debugger code is all updated to be config aware.

Fernando Perez
Owner

I guess it was a matter of being over-conservative in the interest of safety: absent a way to detect the actual user's config choices, going with no colors at least will work OK in all terminals, where as the wrong choice of colors for certain backgrounds produces horrible usability.

So my take would be: if it's not really practical right now to get the standalone debugger/tracer to actually pick up the user's config choices, let's default to nocolor. As long as users can insantiate the tracer themselves with color info, that should suffice for more advanced uses. But I think we should maintain the design principle of plain-but-robust beats pretty-but-brittle.

Min RK
Owner

Okay, then that's how it is now, without changes.

Fernando Perez fperez merged commit aa846e3 into from October 16, 2011
Fernando Perez fperez closed this October 16, 2011
Min RK minrk referenced this pull request October 24, 2011
Closed

debugger.py: pydb broken #636

Fernando Perez fperez referenced this pull request from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 7 unique commits by 1 author.

Oct 14, 2011
Min RK protect IPython from bad custom exception handlers
Previously, errors in custom handlers would result in the custom exception
handler's error being printed in lieu of the real exception, and certain cases could cause infinite loops.

Now, if CustomTB fails it is unregistered immediately, and the original TB is also displayed.

IPython's own BdbQuit_IPython_excepthook had an invalid signature, which revealed this issue, and has also been fixed.

test included.

closes #692
ed5078c
Min RK show tb before entering debug when %pdb is on
closes #690
0d065b1
Min RK prevent atexit handlers from generating crash report
register `sys.excepthook = sys.__excepthook__` with atexit on aplication startup,
so it should be the first handler called.  This removes the crash handler.

closes #207
d5548fa
Oct 15, 2011
Min RK protect against bad return type of CustomTB
includes test
also expands a few docstrings
09803ad
Min RK update set_custom_exc docstring with new-style Parameters section e385f97
Min RK fix IPython check in debugger.Tracer 09c2492
Min RK re-enable pydb flag
Note that pydb has been deprecated, and superseded by pydbgr, which may be
abandoned.

It would be preferable if the Pdb class could inherit from pydb or pdb
based on a runtime flag rather than checking sys.argv at the top level.
This at least restores old behavior for pydb users.

closes #636
d16b38e
This page is out of date. Refresh to see the latest.
4  IPython/core/application.py
@@ -27,6 +27,7 @@
27 27
 # Imports
28 28
 #-----------------------------------------------------------------------------
29 29
 
  30
+import atexit
30 31
 import glob
31 32
 import logging
32 33
 import os
@@ -156,6 +157,9 @@ def init_crash_handler(self):
156 157
         """Create a crash handler, typically setting sys.excepthook to it."""
157 158
         self.crash_handler = self.crash_handler_class(self)
158 159
         sys.excepthook = self.crash_handler
  160
+        def unset_crashhandler():
  161
+            sys.excepthook = sys.__excepthook__
  162
+        atexit.register(unset_crashhandler)
159 163
 
160 164
     def _ipython_dir_changed(self, name, old, new):
161 165
         if old in sys.path:
8  IPython/core/debugger.py
@@ -38,7 +38,7 @@
38 38
 has_pydb = False
39 39
 prompt = 'ipdb> '
40 40
 #We have to check this directly from sys.argv, config struct not yet available
41  
-if '-pydb' in sys.argv:
  41
+if '--pydb' in sys.argv:
42 42
     try:
43 43
         import pydb
44 44
         if hasattr(pydb.pydb, "runl") and pydb.version>'1.17':
@@ -64,7 +64,7 @@ def BdbQuit_excepthook(et,ev,tb):
64 64
     else:
65 65
         BdbQuit_excepthook.excepthook_ori(et,ev,tb)
66 66
 
67  
-def BdbQuit_IPython_excepthook(self,et,ev,tb):
  67
+def BdbQuit_IPython_excepthook(self,et,ev,tb,tb_offset=None):
68 68
     print 'Exiting Debugger.'
69 69
 
70 70
 
@@ -104,8 +104,8 @@ def __init__(self,colors=None):
104 104
         """
105 105
 
106 106
         try:
107  
-            ip = ipapi.get()
108  
-        except:
  107
+            ip = get_ipython()
  108
+        except NameError:
109 109
             # Outside of ipython, we set our own exception hook manually
110 110
             BdbQuit_excepthook.excepthook_ori = sys.excepthook
111 111
             sys.excepthook = BdbQuit_excepthook
104  IPython/core/interactiveshell.py
@@ -1447,29 +1447,37 @@ def set_custom_exc(self, exc_tuple, handler):
1447 1447
 
1448 1448
         Set a custom exception handler, which will be called if any of the
1449 1449
         exceptions in exc_tuple occur in the mainloop (specifically, in the
1450  
-        run_code() method.
  1450
+        run_code() method).
1451 1451
 
1452  
-        Inputs:
  1452
+        Parameters
  1453
+        ----------
  1454
+
  1455
+        exc_tuple : tuple of exception classes
  1456
+            A *tuple* of exception classes, for which to call the defined
  1457
+            handler.  It is very important that you use a tuple, and NOT A
  1458
+            LIST here, because of the way Python's except statement works.  If
  1459
+            you only want to trap a single exception, use a singleton tuple::
1453 1460
 
1454  
-          - exc_tuple: a *tuple* of valid exceptions to call the defined
1455  
-          handler for.  It is very important that you use a tuple, and NOT A
1456  
-          LIST here, because of the way Python's except statement works.  If
1457  
-          you only want to trap a single exception, use a singleton tuple:
  1461
+                exc_tuple == (MyCustomException,)
1458 1462
 
1459  
-            exc_tuple == (MyCustomException,)
  1463
+        handler : callable
  1464
+            handler must have the following signature::
1460 1465
 
1461  
-          - handler: this must be defined as a function with the following
1462  
-          basic interface::
  1466
+                def my_handler(self, etype, value, tb, tb_offset=None):
  1467
+                    ...
  1468
+                    return structured_traceback
1463 1469
 
1464  
-            def my_handler(self, etype, value, tb, tb_offset=None)
1465  
-                ...
1466  
-                # The return value must be
1467  
-                return structured_traceback
  1470
+            Your handler must return a structured traceback (a list of strings),
  1471
+            or None.
1468 1472
 
1469  
-          This will be made into an instance method (via types.MethodType)
1470  
-          of IPython itself, and it will be called if any of the exceptions
1471  
-          listed in the exc_tuple are caught.  If the handler is None, an
1472  
-          internal basic one is used, which just prints basic info.
  1473
+            This will be made into an instance method (via types.MethodType)
  1474
+            of IPython itself, and it will be called if any of the exceptions
  1475
+            listed in the exc_tuple are caught. If the handler is None, an
  1476
+            internal basic one is used, which just prints basic info.
  1477
+
  1478
+            To protect IPython from crashes, if your handler ever raises an
  1479
+            exception or returns an invalid result, it will be immediately
  1480
+            disabled.
1473 1481
 
1474 1482
         WARNING: by putting in your own exception handler into IPython's main
1475 1483
         execution loop, you run a very good chance of nasty crashes.  This
@@ -1478,16 +1486,62 @@ def my_handler(self, etype, value, tb, tb_offset=None)
1478 1486
         assert type(exc_tuple)==type(()) , \
1479 1487
                "The custom exceptions must be given AS A TUPLE."
1480 1488
 
1481  
-        def dummy_handler(self,etype,value,tb):
  1489
+        def dummy_handler(self,etype,value,tb,tb_offset=None):
1482 1490
             print '*** Simple custom exception handler ***'
1483 1491
             print 'Exception type :',etype
1484 1492
             print 'Exception value:',value
1485 1493
             print 'Traceback      :',tb
1486 1494
             #print 'Source code    :','\n'.join(self.buffer)
1487  
-
1488  
-        if handler is None: handler = dummy_handler
1489  
-
1490  
-        self.CustomTB = types.MethodType(handler,self)
  1495
+        
  1496
+        def validate_stb(stb):
  1497
+            """validate structured traceback return type
  1498
+            
  1499
+            return type of CustomTB *should* be a list of strings, but allow
  1500
+            single strings or None, which are harmless.
  1501
+            
  1502
+            This function will *always* return a list of strings,
  1503
+            and will raise a TypeError if stb is inappropriate.
  1504
+            """
  1505
+            msg = "CustomTB must return list of strings, not %r" % stb
  1506
+            if stb is None:
  1507
+                return []
  1508
+            elif isinstance(stb, basestring):
  1509
+                return [stb]
  1510
+            elif not isinstance(stb, list):
  1511
+                raise TypeError(msg)
  1512
+            # it's a list
  1513
+            for line in stb:
  1514
+                # check every element
  1515
+                if not isinstance(line, basestring):
  1516
+                    raise TypeError(msg)
  1517
+            return stb
  1518
+
  1519
+        if handler is None:
  1520
+            wrapped = dummy_handler
  1521
+        else:
  1522
+            def wrapped(self,etype,value,tb,tb_offset=None):
  1523
+                """wrap CustomTB handler, to protect IPython from user code
  1524
+                
  1525
+                This makes it harder (but not impossible) for custom exception
  1526
+                handlers to crash IPython.
  1527
+                """
  1528
+                try:
  1529
+                    stb = handler(self,etype,value,tb,tb_offset=tb_offset)
  1530
+                    return validate_stb(stb)
  1531
+                except:
  1532
+                    # clear custom handler immediately
  1533
+                    self.set_custom_exc((), None)
  1534
+                    print >> io.stderr, "Custom TB Handler failed, unregistering"
  1535
+                    # show the exception in handler first
  1536
+                    stb = self.InteractiveTB.structured_traceback(*sys.exc_info())
  1537
+                    print >> io.stdout, self.InteractiveTB.stb2text(stb)
  1538
+                    print >> io.stdout, "The original exception:"
  1539
+                    stb = self.InteractiveTB.structured_traceback(
  1540
+                                            (etype,value,tb), tb_offset=tb_offset
  1541
+                    )
  1542
+                return stb
  1543
+
  1544
+        self.CustomTB = types.MethodType(wrapped,self)
1491 1545
         self.custom_exceptions = exc_tuple
1492 1546
 
1493 1547
     def excepthook(self, etype, value, tb):
@@ -1556,11 +1610,7 @@ def showtraceback(self,exc_tuple = None,filename=None,tb_offset=None,
1556 1610
                 sys.last_value = value
1557 1611
                 sys.last_traceback = tb
1558 1612
                 if etype in self.custom_exceptions:
1559  
-                    # FIXME: Old custom traceback objects may just return a
1560  
-                    # string, in that case we just put it into a list
1561 1613
                     stb = self.CustomTB(etype, value, tb, tb_offset)
1562  
-                    if isinstance(ctb, basestring):
1563  
-                        stb = [stb]
1564 1614
                 else:
1565 1615
                     if exception_only:
1566 1616
                         stb = ['An exception has occurred, use %tb to see '
@@ -1571,9 +1621,11 @@ def showtraceback(self,exc_tuple = None,filename=None,tb_offset=None,
1571 1621
                         stb = self.InteractiveTB.structured_traceback(etype,
1572 1622
                                                 value, tb, tb_offset=tb_offset)
1573 1623
 
  1624
+                        self._showtraceback(etype, value, stb)
1574 1625
                         if self.call_pdb:
1575 1626
                             # drop into debugger
1576 1627
                             self.debugger(force=True)
  1628
+                        return
1577 1629
 
1578 1630
                 # Actually show the traceback
1579 1631
                 self._showtraceback(etype, value, stb)
8  IPython/core/shellapp.py
@@ -50,6 +50,14 @@
50 50
     "Enable auto calling the pdb debugger after every exception.",
51 51
     "Disable auto calling the pdb debugger after every exception."
52 52
 )
  53
+# pydb flag doesn't do any config, as core.debugger switches on import,
  54
+# which is before parsing.  This just allows the flag to be passed.
  55
+shell_flags.update(dict(
  56
+    pydb = ({},
  57
+        """"Use the third party 'pydb' package as debugger, instead of pdb.
  58
+        Requires that pydb is installed."""
  59
+    )
  60
+))
53 61
 addflag('pprint', 'PlainTextFormatter.pprint',
54 62
     "Enable auto pretty printing of results.",
55 63
     "Disable auto auto pretty printing of results."
34  IPython/core/tests/test_interactiveshell.py
@@ -146,3 +146,37 @@ def test_future_unicode(self):
146 146
         finally:
147 147
             # Reset compiler flags so we don't mess up other tests.
148 148
             ip.compile.reset_compiler_flags()
  149
+
  150
+    def test_bad_custom_tb(self):
  151
+        """Check that InteractiveShell is protected from bad custom exception handlers"""
  152
+        ip = get_ipython()
  153
+        from IPython.utils import io
  154
+        save_stderr = io.stderr
  155
+        try:
  156
+            # capture stderr
  157
+            io.stderr = StringIO()
  158
+            ip.set_custom_exc((IOError,), lambda etype,value,tb: 1/0)
  159
+            self.assertEquals(ip.custom_exceptions, (IOError,))
  160
+            ip.run_cell(u'raise IOError("foo")')
  161
+            self.assertEquals(ip.custom_exceptions, ())
  162
+            self.assertTrue("Custom TB Handler failed" in io.stderr.getvalue())
  163
+        finally:
  164
+            io.stderr = save_stderr
  165
+
  166
+    def test_bad_custom_tb_return(self):
  167
+        """Check that InteractiveShell is protected from bad return types in custom exception handlers"""
  168
+        ip = get_ipython()
  169
+        from IPython.utils import io
  170
+        save_stderr = io.stderr
  171
+        try:
  172
+            # capture stderr
  173
+            io.stderr = StringIO()
  174
+            ip.set_custom_exc((NameError,),lambda etype,value,tb, tb_offset=None: 1)
  175
+            self.assertEquals(ip.custom_exceptions, (NameError,))
  176
+            ip.run_cell(u'a=abracadabra')
  177
+            self.assertEquals(ip.custom_exceptions, ())
  178
+            self.assertTrue("Custom TB Handler failed" in io.stderr.getvalue())
  179
+        finally:
  180
+            io.stderr = save_stderr
  181
+
  182
+
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.