Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

protect IPython from bad custom exception handlers #876

Merged
merged 7 commits into from

2 participants

@minrk
Owner

Previously, errors in custom handlers would result in the custom exception
handler's error being printed in lieu of the real exception, and certain cases could cause infinite loops.

Now, if CustomTB fails it is unregistered immediately, and the original TB is also displayed.

IPython's own BdbQuit_IPython_excepthook had an invalid signature, which revealed this issue, and has also been fixed.

test included.

closes #692

minrk added some commits
@minrk minrk protect IPython from bad custom exception handlers
Previously, errors in custom handlers would result in the custom exception
handler's error being printed in lieu of the real exception, and certain cases could cause infinite loops.

Now, if CustomTB fails it is unregistered immediately, and the original TB is also displayed.

IPython's own BdbQuit_IPython_excepthook had an invalid signature, which revealed this issue, and has also been fixed.

test included.

closes #692
ed5078c
@minrk minrk show tb before entering debug when %pdb is on
closes #690
0d065b1
@minrk
Owner

While looking at the same code, I saw the culprit for #690 as well, so I fixed that, too.

@minrk minrk prevent atexit handlers from generating crash report
register `sys.excepthook = sys.__excepthook__` with atexit on aplication startup,
so it should be the first handler called.  This removes the crash handler.

closes #207
d5548fa
@fperez
Owner

Code looks good, but this simple snippet

from IPython.core.debugger import Tracer
Tracer()

currently exits silently. I've never actually used Tracer() myself, but I'd expect it to activate the debugger in tracing mode, yet it doesn't. Based on the discussion of #692, does this really close the ticket?

@minrk minrk protect against bad return type of CustomTB
includes test
also expands a few docstrings
09803ad
@minrk
Owner

Tracer() is a callable object that can be invoked to set_trace.

I did find a typo to fix, and added return-type validation, which I just pushed.

This is how Tracer appears to be used, based on the source:

from IPython.core.debugger import Tracer
tracer = Tracer()

try:
    1/0
except:
    tracer()
@fperez
Owner

Question, shouln't we be trapping the quit exception? Trying out the example you show, I'm getting this when I quit:

dreamweaver[~]> python mini.py 
--Return--
None
> /home/fperez/mini.py(7)()
      6 except:
----> 7     tracer()
      8 

ipdb> q
Traceback (most recent call last):
  File "mini.py", line 7, in 
    tracer()
  File "/usr/lib/python2.6/bdb.py", line 50, in trace_dispatch
    return self.dispatch_return(frame, arg)
  File "/usr/lib/python2.6/bdb.py", line 84, in dispatch_return
    if self.quitting: raise BdbQuit
bdb.BdbQuit

I have vague memories of, long ago, having written code in ipdb to handle that case, but maybe it's not in the right place when used in standalone tracing mode...

@minrk
Owner

It does handle BdbQuit correctly from IPython. I don't know why it doesn't in regular Python. The exception handler is set when outside IPython here.

Adding a print statement to the excepthook, it is never called.

@minrk
Owner

Nevermind, I found it - ipapi.get() is not an acceptable way to check if you are in IPython - it will always return an IPython shell. get_ipython is the way to check if you are currently in IPython.

@minrk
Owner

Should I also do something about #636, while I'm cleaning up debugger miscellany?

pydb checks for '-pydb' in sys.argv, but there is no pydb flag anymore. Should I add --pydb as a flag to Shell Apps?

@minrk
Owner

Looking into pydb, it would appear that the project is entirely deprecated (files removed from pypi, even), in favor of a rewrite called pydbgr, which is not even installable as far as I can tell, and also appears essentially abandoned.

@minrk minrk re-enable pydb flag
Note that pydb has been deprecated, and superseded by pydbgr, which may be
abandoned.

It would be preferable if the Pdb class could inherit from pydb or pdb
based on a runtime flag rather than checking sys.argv at the top level.
This at least restores old behavior for pydb users.

closes #636
d16b38e
@minrk
Owner

I re-enabled the pydb flag, so it should work. It doesn't set any config, it just allows the flag to pass through, so debugger.py can make its check. Ultimately this should probably be fixed so the check doesn't need to be run at import-time, but for now, at least, we don't have a never-true invalid flag check.

@fperez
Owner

This is looking pretty good. Only one note: 09c2492 makes the correct check, but by not instantiating a full ipython, now the ipdb instance comes up without coloring. It would be nice to restore that, though without waiting to fire up a full blown ipython. That would mean parsing the config for colors...

If you think you can take a quick shot at reading the color settings correctly to instantiate the tracer with color active and matching the user's choice, that would be great.

But if it's looking like too much work, don't sweat it. I don't want to hold this PR forever on that little bit of nicety, so I'm happy to just leave an issue open for color support in the tracer and move on as-is. Your call.

@minrk
Owner
@minrk
Owner

It's not quite as simple as I thought.

There are two defaults:

  • load_default_config() will load your default profile
  • InteractiveShell.colors.get_default_value() will get the default value from the class, sans config.

It's easy to do the latter, but it turns out the former is less easy (or at least less clean), due to Class heirarchy, as we really use TerminalInteractiveShell, a subclass of InteractiveShell, both having a colors argument (again, colors must eventually be removed from the InteractiveShell object, as they are a purely frontend notion, but that's not relevant here).

So, to match the default resolution of the TerminalInteractiveShell's colors with default config would be:

            cfg = load_default_config()
            try:
                def_colors = cfg.TerminalInteractiveShell.colors
            except AttributeError:
                try:
                    def_colors = cfg.InteractiveShell.colors
                except AttributeError:
                    def_colors = InteractiveShell.colors.get_default_value()

And any changes to inheritance could muck this up.

@minrk
Owner

In the debugger code, the default colors are hardcoded to NoColor (Tracer has a colors parameter, while Pdb has color_scheme), is it possible that's deliberate and/or desirable? I can easily change them to use InteractiveShell's default. I think using config files should be put off until the debugger code is all updated to be config aware.

@fperez
Owner

I guess it was a matter of being over-conservative in the interest of safety: absent a way to detect the actual user's config choices, going with no colors at least will work OK in all terminals, where as the wrong choice of colors for certain backgrounds produces horrible usability.

So my take would be: if it's not really practical right now to get the standalone debugger/tracer to actually pick up the user's config choices, let's default to nocolor. As long as users can insantiate the tracer themselves with color info, that should suffice for more advanced uses. But I think we should maintain the design principle of plain-but-robust beats pretty-but-brittle.

@minrk
Owner

Okay, then that's how it is now, without changes.

@fperez fperez merged commit aa846e3 into from
@fperez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 14, 2011
  1. @minrk

    protect IPython from bad custom exception handlers

    minrk authored
    Previously, errors in custom handlers would result in the custom exception
    handler's error being printed in lieu of the real exception, and certain cases could cause infinite loops.
    
    Now, if CustomTB fails it is unregistered immediately, and the original TB is also displayed.
    
    IPython's own BdbQuit_IPython_excepthook had an invalid signature, which revealed this issue, and has also been fixed.
    
    test included.
    
    closes #692
  2. @minrk
  3. @minrk

    prevent atexit handlers from generating crash report

    minrk authored
    register `sys.excepthook = sys.__excepthook__` with atexit on aplication startup,
    so it should be the first handler called.  This removes the crash handler.
    
    closes #207
Commits on Oct 15, 2011
  1. @minrk

    protect against bad return type of CustomTB

    minrk authored
    includes test
    also expands a few docstrings
  2. @minrk
  3. @minrk
  4. @minrk

    re-enable pydb flag

    minrk authored
    Note that pydb has been deprecated, and superseded by pydbgr, which may be
    abandoned.
    
    It would be preferable if the Pdb class could inherit from pydb or pdb
    based on a runtime flag rather than checking sys.argv at the top level.
    This at least restores old behavior for pydb users.
    
    closes #636
This page is out of date. Refresh to see the latest.
View
4 IPython/core/application.py
@@ -27,6 +27,7 @@
# Imports
#-----------------------------------------------------------------------------
+import atexit
import glob
import logging
import os
@@ -156,6 +157,9 @@ def init_crash_handler(self):
"""Create a crash handler, typically setting sys.excepthook to it."""
self.crash_handler = self.crash_handler_class(self)
sys.excepthook = self.crash_handler
+ def unset_crashhandler():
+ sys.excepthook = sys.__excepthook__
+ atexit.register(unset_crashhandler)
def _ipython_dir_changed(self, name, old, new):
if old in sys.path:
View
8 IPython/core/debugger.py
@@ -38,7 +38,7 @@
has_pydb = False
prompt = 'ipdb> '
#We have to check this directly from sys.argv, config struct not yet available
-if '-pydb' in sys.argv:
+if '--pydb' in sys.argv:
try:
import pydb
if hasattr(pydb.pydb, "runl") and pydb.version>'1.17':
@@ -64,7 +64,7 @@ def BdbQuit_excepthook(et,ev,tb):
else:
BdbQuit_excepthook.excepthook_ori(et,ev,tb)
-def BdbQuit_IPython_excepthook(self,et,ev,tb):
+def BdbQuit_IPython_excepthook(self,et,ev,tb,tb_offset=None):
print 'Exiting Debugger.'
@@ -104,8 +104,8 @@ def __init__(self,colors=None):
"""
try:
- ip = ipapi.get()
- except:
+ ip = get_ipython()
+ except NameError:
# Outside of ipython, we set our own exception hook manually
BdbQuit_excepthook.excepthook_ori = sys.excepthook
sys.excepthook = BdbQuit_excepthook
View
104 IPython/core/interactiveshell.py
@@ -1447,29 +1447,37 @@ def set_custom_exc(self, exc_tuple, handler):
Set a custom exception handler, which will be called if any of the
exceptions in exc_tuple occur in the mainloop (specifically, in the
- run_code() method.
+ run_code() method).
- Inputs:
+ Parameters
+ ----------
+
+ exc_tuple : tuple of exception classes
+ A *tuple* of exception classes, for which to call the defined
+ handler. It is very important that you use a tuple, and NOT A
+ LIST here, because of the way Python's except statement works. If
+ you only want to trap a single exception, use a singleton tuple::
- - exc_tuple: a *tuple* of valid exceptions to call the defined
- handler for. It is very important that you use a tuple, and NOT A
- LIST here, because of the way Python's except statement works. If
- you only want to trap a single exception, use a singleton tuple:
+ exc_tuple == (MyCustomException,)
- exc_tuple == (MyCustomException,)
+ handler : callable
+ handler must have the following signature::
- - handler: this must be defined as a function with the following
- basic interface::
+ def my_handler(self, etype, value, tb, tb_offset=None):
+ ...
+ return structured_traceback
- def my_handler(self, etype, value, tb, tb_offset=None)
- ...
- # The return value must be
- return structured_traceback
+ Your handler must return a structured traceback (a list of strings),
+ or None.
- This will be made into an instance method (via types.MethodType)
- of IPython itself, and it will be called if any of the exceptions
- listed in the exc_tuple are caught. If the handler is None, an
- internal basic one is used, which just prints basic info.
+ This will be made into an instance method (via types.MethodType)
+ of IPython itself, and it will be called if any of the exceptions
+ listed in the exc_tuple are caught. If the handler is None, an
+ internal basic one is used, which just prints basic info.
+
+ To protect IPython from crashes, if your handler ever raises an
+ exception or returns an invalid result, it will be immediately
+ disabled.
WARNING: by putting in your own exception handler into IPython's main
execution loop, you run a very good chance of nasty crashes. This
@@ -1478,16 +1486,62 @@ def my_handler(self, etype, value, tb, tb_offset=None)
assert type(exc_tuple)==type(()) , \
"The custom exceptions must be given AS A TUPLE."
- def dummy_handler(self,etype,value,tb):
+ def dummy_handler(self,etype,value,tb,tb_offset=None):
print '*** Simple custom exception handler ***'
print 'Exception type :',etype
print 'Exception value:',value
print 'Traceback :',tb
#print 'Source code :','\n'.join(self.buffer)
-
- if handler is None: handler = dummy_handler
-
- self.CustomTB = types.MethodType(handler,self)
+
+ def validate_stb(stb):
+ """validate structured traceback return type
+
+ return type of CustomTB *should* be a list of strings, but allow
+ single strings or None, which are harmless.
+
+ This function will *always* return a list of strings,
+ and will raise a TypeError if stb is inappropriate.
+ """
+ msg = "CustomTB must return list of strings, not %r" % stb
+ if stb is None:
+ return []
+ elif isinstance(stb, basestring):
+ return [stb]
+ elif not isinstance(stb, list):
+ raise TypeError(msg)
+ # it's a list
+ for line in stb:
+ # check every element
+ if not isinstance(line, basestring):
+ raise TypeError(msg)
+ return stb
+
+ if handler is None:
+ wrapped = dummy_handler
+ else:
+ def wrapped(self,etype,value,tb,tb_offset=None):
+ """wrap CustomTB handler, to protect IPython from user code
+
+ This makes it harder (but not impossible) for custom exception
+ handlers to crash IPython.
+ """
+ try:
+ stb = handler(self,etype,value,tb,tb_offset=tb_offset)
+ return validate_stb(stb)
+ except:
+ # clear custom handler immediately
+ self.set_custom_exc((), None)
+ print >> io.stderr, "Custom TB Handler failed, unregistering"
+ # show the exception in handler first
+ stb = self.InteractiveTB.structured_traceback(*sys.exc_info())
+ print >> io.stdout, self.InteractiveTB.stb2text(stb)
+ print >> io.stdout, "The original exception:"
+ stb = self.InteractiveTB.structured_traceback(
+ (etype,value,tb), tb_offset=tb_offset
+ )
+ return stb
+
+ self.CustomTB = types.MethodType(wrapped,self)
self.custom_exceptions = exc_tuple
def excepthook(self, etype, value, tb):
@@ -1556,11 +1610,7 @@ def showtraceback(self,exc_tuple = None,filename=None,tb_offset=None,
sys.last_value = value
sys.last_traceback = tb
if etype in self.custom_exceptions:
- # FIXME: Old custom traceback objects may just return a
- # string, in that case we just put it into a list
stb = self.CustomTB(etype, value, tb, tb_offset)
- if isinstance(ctb, basestring):
- stb = [stb]
else:
if exception_only:
stb = ['An exception has occurred, use %tb to see '
@@ -1571,9 +1621,11 @@ def showtraceback(self,exc_tuple = None,filename=None,tb_offset=None,
stb = self.InteractiveTB.structured_traceback(etype,
value, tb, tb_offset=tb_offset)
+ self._showtraceback(etype, value, stb)
if self.call_pdb:
# drop into debugger
self.debugger(force=True)
+ return
# Actually show the traceback
self._showtraceback(etype, value, stb)
View
8 IPython/core/shellapp.py
@@ -50,6 +50,14 @@
"Enable auto calling the pdb debugger after every exception.",
"Disable auto calling the pdb debugger after every exception."
)
+# pydb flag doesn't do any config, as core.debugger switches on import,
+# which is before parsing. This just allows the flag to be passed.
+shell_flags.update(dict(
+ pydb = ({},
+ """"Use the third party 'pydb' package as debugger, instead of pdb.
+ Requires that pydb is installed."""
+ )
+))
addflag('pprint', 'PlainTextFormatter.pprint',
"Enable auto pretty printing of results.",
"Disable auto auto pretty printing of results."
View
34 IPython/core/tests/test_interactiveshell.py
@@ -146,3 +146,37 @@ def test_future_unicode(self):
finally:
# Reset compiler flags so we don't mess up other tests.
ip.compile.reset_compiler_flags()
+
+ def test_bad_custom_tb(self):
+ """Check that InteractiveShell is protected from bad custom exception handlers"""
+ ip = get_ipython()
+ from IPython.utils import io
+ save_stderr = io.stderr
+ try:
+ # capture stderr
+ io.stderr = StringIO()
+ ip.set_custom_exc((IOError,), lambda etype,value,tb: 1/0)
+ self.assertEquals(ip.custom_exceptions, (IOError,))
+ ip.run_cell(u'raise IOError("foo")')
+ self.assertEquals(ip.custom_exceptions, ())
+ self.assertTrue("Custom TB Handler failed" in io.stderr.getvalue())
+ finally:
+ io.stderr = save_stderr
+
+ def test_bad_custom_tb_return(self):
+ """Check that InteractiveShell is protected from bad return types in custom exception handlers"""
+ ip = get_ipython()
+ from IPython.utils import io
+ save_stderr = io.stderr
+ try:
+ # capture stderr
+ io.stderr = StringIO()
+ ip.set_custom_exc((NameError,),lambda etype,value,tb, tb_offset=None: 1)
+ self.assertEquals(ip.custom_exceptions, (NameError,))
+ ip.run_cell(u'a=abracadabra')
+ self.assertEquals(ip.custom_exceptions, ())
+ self.assertTrue("Custom TB Handler failed" in io.stderr.getvalue())
+ finally:
+ io.stderr = save_stderr
+
+
Something went wrong with that request. Please try again.