Fix #481 using custom qt4 input hook #815

merged 15 commits into from Oct 28, 2011

4 participants


This is my tentative fix for issue #481, as discussed in #481 (comment).

This is not meant to be merged as such, as I only could provide limited testing:

  • seems to fix the issue for Windows (at least Win 7 x64)
  • no idea about the impact on Linux, Mac; at best it will also fix the issue for Mac ;-)
  • couldn't check if the refactoring done in the first cset 3d1dca8 didn't break Wx support (I don't have it installed)

Also, I've currently inlined the hook, as it was small enough it didn't feel appropriate to create a IPython/lib/ file.

Finally, the behavior with Ctrl-C is currently quite catastrophic, though this is not related to my changes, as it's the same behavior with the original 0.11 install, even without Qt involved. But this probably another topic.

See second iteration for the merge candidate.

@takluyver takluyver and 1 other commented on an outdated diff Sep 22, 2011
import sys
import warnings
+if == 'posix':
+ import select
+elif sys.platform == 'win32':
takluyver Sep 22, 2011 IPython member

Are there cases where neither of these are true? What would happen if so?

Also, a minor nitpick, is it necessary to use for one test and sys.platform for the other? Can we get the information we need from just one?

takluyver Sep 22, 2011 IPython member

(OK, I see now this code is just moved from another file. It's still worth thinking about while we're looking at it, though)

cboos Sep 22, 2011

Well, you have the same if/elif test structure in the stdin_ready() function, so it will just work.
Perhaps it would be cleaner to move those imports in the stdin_ready() function directly?
(if that's not against your coding conventions)

takluyver Sep 22, 2011 IPython member

But if neither of these conditions were true, stdin_ready will always return None, which looks like it would put the code below into an infinite loop (I could be wrong, I don't do much GUI stuff).

cboos Sep 23, 2011

Correct! I don't think there's a supported platform which doesn't fall in either category, but anyway returning True as a fallback seems safer. My next iteration of the function will look like this:

def stdin_ready():
    if == 'posix':
        import select
        infds, outfds, erfds =[sys.stdin],[],[],0)
        if infds:
            return True
            return False
    elif sys.platform == 'win32':
        import msvcrt
        return msvcrt.kbhit()
    return True # assume there's something so that we won't wait forever

Also, I assume that the original combination of was probably done on purpose: match Linux and Mac with the first test, and pick non-cygwin python on Windows with the second test.

takluyver Sep 23, 2011 IPython member

What are and sys.platform inside cygwin? Does the function work properly in this case too?

cboos Sep 23, 2011

For cygwin, is 'posix' and sys.platform is 'cygwin', so Cygwin is handled by the first case, environments which have select.
We could replace the second test with simply == 'nt' if you prefer, the semantic will be the same if the tests are done in that order.

(PS: I haven't tested in cygwin, I don't even know if it's a supported platform!)

takluyver Sep 23, 2011 IPython member

If testing for == 'nt' will behave as expected, I think that's better than using two different variables to test. I've no idea if IPython can run on Cygwin, but if it can, I don't doubt there'll be someone using it.

FWIW, the Python docs describe sys.platform as having a "finer granularity" than

@takluyver takluyver and 1 other commented on an outdated diff Sep 22, 2011
if app is None:
app = QtCore.QCoreApplication.instance()
if app is None:
app = QtGui.QApplication([" "])
+ if 'pyreadline' in sys.modules:
takluyver Sep 22, 2011 IPython member

This looks like we no longer do anything if pyreadline isn't in use (i.e. on any Linux or Mac platforms). Don't we need the pyqtRestoreInputHook() call in somewhere for the other platforms?

cboos Sep 22, 2011

Ah right. But actually what needs to be done here is to also activate the inputhook_qt4 for Mac, and probably for Linux as well (I think it wouldn't hurt either but I'll verify).

Without any readline package, the input hook gets called as well, but much less frequently: only once before waiting for a new complete input, not once after every character. That's why the delay induced by the default PyQt4 hook was not so noticeable in that situation.

So maybe we just need to remove that test altogether. If someone could verify the next changeset on a Mac, that would be great!


Dealing with CTRL+C proves to be quite uneasy. First on Windows with Python 2.7, I see all the weird effects described in #625. The root cause might well be a Python bug, see e.g. though in my case the python interpreter exits immediately, at the first CTRL+C).

So here, at the very least we need to trap KeyboardInterrupt in the hook, like it's done for other gui backends (6d5890c).

In 0dd489f, I also played with the idea of breaking the event loop processing with a first press to CTRL+C, which lets a second CTRL+C raise a proper KeyboardInterrupt and makes you return to the prompt. At this point the event processing is restored in the inputhook thanks to a pre_prompt_hook. I find this behavior to be quite OK (kind of fix for #122).

But that only works when you start IPython with python.exe!
If instead you run ipython.exe, then no matter what, a CTRL+C will kill the ipython.exe process and leaves its python.exe subprocess and the shell to fight for the stdin...
If the shell is the cmd.exe, it will grab all the input until you "exit", at which point the python.exe will regain control.
If the shell is msys bash, then you have the impression that bash and the remaining python.exe get each a bit of stdin...
Hope my report is not too confusing, because the situation in the console really is ;-)

What I ideally would like to achieve is the 0dd489f behavior but also when starting IPython via ipython.exe.
Btw. any hints where to look for the ipython.exe -> python.exe code or design rationale? I suppose there must be a reason why there is a subprocess...

IPython member

I'm not a Windows user, so I can't help you with the specifics there. I think ipython.exe is something created by Enthought for EPD, so you might want to get in touch with them for how that's put together.

IPython member

Hey guys, just a quick note: I haven't followed the PR, but right now it doesn't merge cleanly anymore. Sorry about that, there's been enough churn on master lately that some PRs fell behind.

So this one needs a rebase before we can proceed further, let us know if you need a hand with that.

Unless you guys have decided you want to take a different approach altogether with a new PR, I'll leave that up to you. I just want to give you a heads-up b/c I'm starting to map out the landscape to plan for an 0.12 release, and un-mergeable PRs will be obviously an issue there.

IPython member

I think a big PR that I merged carelessly may have caused various conflicts, so I'm happy to lend a hand rebasing if you like.


I wanted to rework the changesets anyway to take the feedback into account, so I don't mind rebasing. I'll be able to create a new PR this week-end (or is there a way to associate a second branch or a rebased branch on the same PR?).

IPython member

It's a bit 'impure', but if you rebase a branch and force-push it (i.e. push the new branch with the old name), it updates the pull request accordingly. We do that quite a bit.

IPython member
cboos added some commits Sep 21, 2011
@cboos cboos inputhook: make stdin_ready() function reusable
Move it from lib.inputhookwx to lib.inputhook, as the former
module can only be loaded if wx itself is available.
@cboos cboos inputhook: improve stdin_ready()
 - move select and msvcrt imports in the function itself
   as they're only needed there
 - switch on for platform dependent code
 - for unknow platforms, assume there's something to read
@cboos cboos inputhook: make PyQt4 plays nicer with pyreadline
PyQt4's own input hook is not really appropriate when PyReadline is active
(and possibly other readline packages running the PyOS_InputHook repeatedly).
Arguably this should be fixed in PyQt4 itself, but in the meantime we can
install a local hook which does the Qt event loop processing in a less
aggressive way.
@cboos cboos inputhook: improve CTRL+C handling with qt4
Allow a first CTRL+C to interrupt the qt4 event loop
and a second to trigger a KeyboardInterrupt.
@cboos cboos inputhook_qt4: handle KeyboardInterrupt in a way compatible with 'rea…

While the previous code works fine on Windows, on Linux with
'readline' the situation is different: once we disable the Qt4 event
loop, we go back waiting in the select() inside readline.c's
readline_until_enter_or_signal() function. When that select() is
interrupted, the PyOS_InputHook is executed if present and the pending
KeyboardInterrupt is delivered as soon as bytecode execution starts,
even before entering the `try:` clause! As at that point we're in a
ctypes callback, this will lead to an internal error. The only
solution is therefore to disable the PyOS_InputHook temporarily.

Second iteration of the patchset.

The first 3 changesets deal with #481:

  • 42c877c simply moves stdin_ready() from to
  • a761793 improves stdin_ready() according to the feedback from takluyver
  • 76f9a74 uses a custom inputhook instead of PyQt's qtcore_input_hook

The last 2 changesets deal with the Qt part of #122:

  • 6ab38c6 allows a first CTRL+C to break the event loop and a second to clear the prompt (at which point event processing resumes)
  • fa66348 obtaining the same behavior on Linux needs a slightly different approach

The last two are not needed to fix #481, but without them the PyQt4 support seems buggy, as explained in #122.

As explained in the changesets themselves, I think there are a few technical limitations in ctypes that prevent a single CTRL+C to work as expected, hence the "two CTRL+C" approach which I believe is intuitive enough, even more so now that I print after the first CTRL+C:

(event loop interrupted - hit CTRL+C again to clear the prompt)

Open points:

  1. not tested on Mac
  2. on Linux, an auto-repeat of CTRL+C pressed down leads to the same non-fatal internal error as the one we get without fa66348; (experiment with signal()?)
  3. on Windows, CTRL+C is only dealt with correctly in a cmd.exe shell when running python, all the other combinations are still problematic (as per #625)

I think only 1. prevents merging, the others could correspond to later improvements.

@fperez fperez commented on an outdated diff Oct 9, 2011
+def stdin_ready():
fperez Oct 9, 2011 IPython member

All functions should have a docstring. The full docstring guidelines follow the numpy standard, but for such a small function probably a single one-liner suffices.

Though in this case, I'd prefer it if the function was split in three: one for posix, one for nt, and an 'others'. The if statement that's currently inside would then instead assign the public name to the right implementation. This raises slightly the import cost, but means we're not doing platform checks on every single call.

@fperez fperez commented on an outdated diff Oct 9, 2011
if app is None:
app = QtCore.QCoreApplication.instance()
if app is None:
app = QtGui.QApplication([" "])
+ # Always use a custom input hook instead of PyQt4's default
+ # one, as it interacts better with readline packages (issue
+ # #481).
+ # Note that we can't let KeyboardInterrupt escape from that
+ # hook, as no exception can be raised from within a ctypes
+ # python callback. We need to make a compromise: a trapped
+ # KeyboardInterrupt will temporarily disable the input hook
+ # until we start over with a new prompt line with a second
+ # CTRL+C.
+ got_kbdint = [False]
+ def inputhook_qt4():
fperez Oct 9, 2011 IPython member

Nested functions that are starting to get this long make the code harder to read. I'd suggest moving it to the top-level, make it private with a leading underscore if need be (but please drop in a docstring regardless; we're really trying to ensure there's docstrings everywhere).

IPython member

Thanks a lot for the updates. I had some inline comments, once those are handled we'll play further with this guy, it looks like we're getting close to being able to merge it, and it should be in time for 0.12.

cboos added some commits Oct 9, 2011
@cboos cboos inputhook: further cleanups for stdin_ready()
 - split stdin_ready() in 3 platform specific functions and choose
   the appropriate one at import time
 - add docstrings for each of these
@cboos cboos inputhook: move inputhook_qt4 related code in own file aa6d06b

I agree that enable_qt4() was a bit too long and needed clean-up.

However, I think it makes sense to keep inputhook_qt4() and preprompthook_qt4() as subfunctions, so that they can share a variable from a parent function create_inputhook_qt4(). I extracted that function into a separate file, like there's one already for most other GUIs.

cboos added some commits Oct 9, 2011
@cboos cboos inputhookqt4: forgot to reset KeyboardInterrupt flag c795b61
@cboos cboos inputhookqt4: polish the qt4 related hooks
 - make sure we don't install more than one 'pre_prompt_hook'
   for a given ipapi instance
 - clarify the internal documentation and move it from comments
   into docstrings

I was a bit worried about the use of a 'pre_prompt_hook', as I was not completely sure about the lifetime and scope of the objects involved (is ipapi.get() a singleton?). With e3612ec at least, we can safely have multiple sequences of enable_gui('qt4') / enable_gui(None), should that ever be needed ;-)

Btw, it seems that there's no way to get enable_gui(None) via the %gui. Should we add GUI_NONE = 'none'?

IPython member

While working on the open point 2. auto-repeat of CTRL+C leads to an internal error, I found a solution which implies some cooperation between set_inputhook, clear_inputhook and the actual callback (inputhook_qt4). It works fine, I no longer have the internal error, but the problem is that the other GUIs will not see a KeyboardInterrupt anymore. It's not that critical in the sense they were just ignoring it anyway or not handling it at all (cf. #122, #825), but to me this means we can probably do better and extend my two step approach (first CTRL+C to disable the event loop, second to trigger a normal KeyboardInterrupt) to all GUIs, not just Qt...

I think this will lead to a cleaner approach overall.

If in the meantime someone could test the current state on Mac, this would be interesting as well ;-)

@cboos cboos inputhook: disable CTRL+C when a hook is active.
On systems with 'readline', it's very likely to intercept a signal
during a select() call. The default SIGINT handler will schedule a
KeyboardInterrupt exception to be raised as soon as possible. If
ctypes is used to install a Python callback for PyOS_InputHook, this
will happen as soon as the bytecode execution starts, so even if the
first instruction of the callback is a
`try: ... except KeyboardInterrupt` clause, it's actually too late.

As ctypes doesn't allow a Python callback to raise an exception, this
ends up with IPython detecting an internal error... not pretty.  We
must therefore ignore the SIGINT signals until we are sure the
exception handler is active, in the Python callback.

Okay... so here it goes ;-)

The last two commits lead to a result I'm much more satisfied with. It also feels more reusable for other GUIs. They are normally not affected by the change, as the set_inputhook() keeps the current semantic and I've introduced a set_safe_inputhook for the new behavior, so far only used for qt4.

If you look at this PR now, you may want to skip most of the intermediate changes and look at the final and files, I think I've documented everything quite clearly ;-)

If you like the approach, I could try to rebase once more and go to this approach directly.


Sorry, just noticed a missing line in the commit message above, the end of the 2nd paragraph should read:

This should help to address issues #122 and #825 (however some
cooperation from the input hook is needed, in particular it should not

catch KeyboardInterrupt and has to be installed using

@fperez fperez and 1 other commented on an outdated diff Oct 14, 2011
+ except KeyboardInterrupt:
+ ignore_CTRL_C()
+ self._inputhook_on_hold = safe_callback
+ print("\nKeyboardInterrupt - event loop interrupted!"
+ "\n * hit CTRL+C again to return to the prompt"
+ "\n (event loop will then be resumed)"
+ "\n * use '%%gui none' to disable the event loop"
+ " permanently"
+ "\n and '%%gui %s' to re-enable it later\n" %
+ (self._current_gui or '...'))
+ self.suspend_inputhook()
+ return 0
+ safe_callback.wrapped = callback
+ # register _restore_inputhook() method as a 'pre_prompt_hook' (once)
+ ip = get_ipython()
fperez Oct 14, 2011 IPython member

Note that this doesn't actually work: the get_ipython function is a utility function meant for user code that is only available at the interactive prompt once IPython itself is fully running. You can, however, use this:

from IPython.core.interactiveshell import InteractiveShell
ip = InteractiveShell.instance()

As a general note, for this PR you should make sure that in addition to the automated test suite, you validate things by manually starting ipython with --gui X for all valid values of X, and also run with --pylab and all the valid pylab GUIs, as well as the gui examples in docs/examples/lib:*****

Sorry to ask all this, but since event loop integration is inherently an interactive problem, we need to ensure it's all tested interactively with all the various toolkits, so there's no easy shortcut I know of besides actually running each example with its corresponding toolkit.

takluyver Oct 14, 2011 IPython member

Just to ask: is it a good idea to add code using InteractiveShell.instance()? I thought we were trying to move away from InteractiveShell being a singleton.

fperez Oct 17, 2011 IPython member

Well, but right now I don't think we have an alternate mechanism for accessing the currently running instance, and code such as this that does indeed need to get to it has to use something. get_ipython is something we inject into the user's namespace, so for now I don't see an alternative.

It's true that in the long run we do want to move away from that signleton design, but that's probably going to take deeper refactoring than we can tackle right now. At least the instance() call is a single, well-labeled entry point that will be easy to search for when we refactor things.

@cboos, just a heads-up: we're trying to wrap up the in-flight PRs over the next couple of weeks. Let us know if you think you'll have a chance to work on it over the next few days, so we can get it in for the 0.12 release. Thanks!


@fperez: I understand that when I'm about to modify the whole logic about lib/inputhook I should also test the different GUI backends... but I see no chance to do that in the coming weeks. I've set up a new development environment where I will hopefully be able to test PySide in addition to PyQt and maybe Tk and Gtk, but that's about it for now.

So what I propose is to leave the more ambitious change 1d6e9a0 for a later version, but finish 0fc80df (the next to last changeset, i.e the first simpler approach which doesn't refactor by adding a missing ignore_CTRL_C() and fixing the get_ipython() stuff. That sequence of changes only touches the qt4 related code and has no impact on the other GUIs. I should probably rebase that once more for increased clarity.

About fixing get_ipython() fixing, I'm asking again just to be sure: what is the best approach, reverting 350ecdb (i.e. use IPython.core.ipapi.get()) or use IPython.core.interactiveshell.InteractiveShell.instance()?

And still, even by leaving 1d6e9a0 aside, another blocking factor I see for the merge is that someone needs to test this on a Mac, as I can't do that myself.

IPython member
IPython member

Looks pretty good on OSX+PySide, except for at exit, I get an infinite crash loop:
The first error:

Do you really want to exit ([y]/n)? 
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/Users/minrk/dev/py/matplotlib/lib/matplotlib/", line 82, in destroy_all
  File "/Users/minrk/dev/py/matplotlib/lib/matplotlib/backends/", line 388, in destroy
    self._widgetclosed )
RuntimeError: Internal C++ object (PySide.QtGui.QMainWindow) already deleted.
Error in sys.exitfunc:
RuntimeError                                  Python 2.7.1: /usr/bin/python
                                                   Fri Oct 21 14:29:14 2011
A problem occured executing Python code.  Here is the sequence of function
calls leading up to the error, with the most recent (innermost) call last.
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/atexit.pyc in _run_exitfuncs()
     10 import sys
     12 _exithandlers = []
     13 def _run_exitfuncs():
     14     """run any registered exit functions
     16     _exithandlers is traversed in reverse order so functions are executed
     17     last in, first out.
     18     """
     20     exc_info = None
     21     while _exithandlers:
     22         func, targs, kargs = _exithandlers.pop()
     23         try:
---> 24             func(*targs, **kargs)
        func = 
        global function = undefined
        global to = undefined
        global be = undefined
        global called = undefined
        global at = undefined
        global exit = undefined
     25         except SystemExit:
     26             exc_info = sys.exc_info()
     27         except:
     28             import traceback
     29             print >> sys.stderr, "Error in atexit._run_exitfuncs:"
     30             traceback.print_exc()
     31             exc_info = sys.exc_info()
     33     if exc_info is not None:
     34         raise exc_info[0], exc_info[1], exc_info[2]
     37 def register(func, *targs, **kargs):
     38     """register a function to be executed upon normal program termination

/Users/minrk/dev/py/matplotlib/lib/matplotlib/_pylab_helpers.pyc in destroy_all()
     67         #print len(Gcf.figs.keys()), len(Gcf._activeQue)
     68         manager.destroy()
     69         gc.collect()
     71     @staticmethod
     72     def destroy_fig(fig):
     73         "*fig* is a Figure instance"
     74         for manager in Gcf.figs.values():
     75             if manager.canvas.figure == fig:
     76                 Gcf.destroy(manager.num)
     78     @staticmethod
     79     def destroy_all():
     80         for manager in Gcf.figs.values():
     81             manager.canvas.mpl_disconnect(manager._cidgcf)
---> 82             manager.destroy()
     84         Gcf._activeQue = []
     85         Gcf.figs.clear()
     86         gc.collect()
     88     @staticmethod
     89     def has_fignum(num):
     90         """
     91         Return *True* if figure *num* exists.
     92         """
     93         return num in Gcf.figs
     95     @staticmethod
     96     def get_all_fig_managers():
     97         """

/Users/minrk/dev/py/matplotlib/lib/matplotlib/backends/backend_qt4.pyc in destroy(self=, *args=())
    373         else:
    374             toolbar = None
    375         return toolbar
    377     def resize(self, width, height):
    378         'set the canvas size in pixels'
    379         self.window.resize(width, height)
    381     def show(self):
    384     def destroy( self, *args ):
    385         if self.window._destroying: return
    386         self.window._destroying = True
    387         QtCore.QObject.disconnect( self.window, QtCore.SIGNAL( 'destroyed()' ),
--> 388                                    self._widgetclosed )
    389         if self.toolbar: self.toolbar.destroy()
    390         if DEBUG: print "destroy figure manager"
    391         self.window.close()
    393     def set_window_title(self, title):
    394         self.window.setWindowTitle(title)
    396 class NavigationToolbar2QT( NavigationToolbar2, QtGui.QToolBar ):
    397     def __init__(self, canvas, parent, coordinates=True):
    398         """ coordinates: should we show the coordinates on the right? """
    399         self.canvas = canvas
    400         self.coordinates = coordinates
    401         QtGui.QToolBar.__init__( self, parent )
    402         NavigationToolbar2.__init__( self, canvas )

RuntimeError: Internal C++ object (PySide.QtGui.QMainWindow) already deleted.

Which may be a PySide (I'm using 1.06) or matplotlib (git master) bug, though it does not come up in IPython master, so something in this PR is relevant.

But the real problem is that this means that QtCore.QCoreApplication.instance() returns None after shutdown, and the crash handler gets itself stuck permanently in this loop:

Error in sys.excepthook:
Traceback (most recent call last):
  File "/Users/minrk/dev/ip/mine/IPython/core/", line 159, in __call__
    raw_input("Hit  to quit this message (your terminal may close):")
RuntimeError: can't re-enter readline

Original exception was:
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 313, in 'calling callback function'
  File "/Users/minrk/dev/ip/mine/IPython/lib/", line 84, in inputhook_qt4
    app.processEvents(QtCore.QEventLoop.AllEvents, 300)
AttributeError: 'NoneType' object has no attribute 'processEvents'
AttributeError                                Python 2.7.1: /usr/bin/python
                                                   Fri Oct 21 14:29:15 2011
A problem occured executing Python code.  Here is the sequence of function
calls leading up to the error, with the most recent (innermost) call last.
/Users/minrk/dev/ip/mine/docs/_ctypes/callbacks.c in 'calling callback function'()

/Users/minrk/dev/ip/mine/IPython/lib/inputhookqt4.pyc in inputhook_qt4()
     69     def inputhook_qt4():
     70         """PyOS_InputHook python hook for Qt4.
     72         Process pending Qt events and if there's no pending keyboard
     73         input, spend a short slice of time (50ms) running the Qt event
     74         loop.
     76         As a Python ctypes callback can't raise an exception, we catch
     77         the KeyboardInterrupt and temporarily deactivate the hook,
     78         which will let a *second* CTRL+C be processed normally and go
     79         back to a clean prompt line.
     80         """
     81         try:
     82             allow_CTRL_C()
     83             app = QtCore.QCoreApplication.instance()
---> 84             app.processEvents(QtCore.QEventLoop.AllEvents, 300)
     85             if not stdin_ready():
     86                 timer = QtCore.QTimer()
     87                 timer.timeout.connect(app.quit)
     88                 while not stdin_ready():
     89                     timer.start(50)
     90                     app.exec_()
     91                     timer.stop()
     92             ignore_CTRL_C()
     93         except KeyboardInterrupt:
     94             ignore_CTRL_C()
     95             got_kbdint[0] = True
     96             print("\nKeyboardInterrupt - qt4 event loop interrupted!"
     97                   "\n  * hit CTRL+C again to clear the prompt"
     98                   "\n  * use '%gui none' to disable the event loop"
     99                   " permanently"

AttributeError: 'NoneType' object has no attribute 'processEvents'

where the inputhook raises an error, and the crash handler invokes the inputhook for its Hit <Enter>... message, repeating forever.

This points to a general issue - when we inject a custom inputhook, they should probably all be wrapped in a simple try/finally that restores the original, so that errors don't repeat infinitely.


Thanks for testing on Mac!

Both the original error and the one in my hook have a common cause: assuming the qApp is still alive, which for some reason is not the case here. I'll try to reproduce that issue.

I can check for this situation in the qt4 hook and as you suggested make sure that the hook will unregister itself in case of an error.

Also in parallel to this set of changes, I continued the work started in 1d6e9a0 for using the two step CTRL+C approach for all GUIs, and some initial testing has shown that this approach can also work well for gtk and Tk. When this is ready I'll create a second PR in order to keep this one focused on the immediate problem with Qt4 (i.e. fixing #481).

IPython member

That makes sense. Can you see why the atexit behavior is not the same with the new code?


I installed PySide on Windows (the latest 1.0.7 installer) and I could reproduce the problem... only when I mistakenly mixed PyQt4 and PySide code! They seem to both share the same instances of Qt libraries and to interfere badly at exit. Are you sure you don't have a from PyQt4 import ... statement somewhere? Or maybe that mixup is in matplotlib, as some other people also noticed a similar issue when using PySide with matplotlib (matplotlib/matplotlib#80 (comment)).

If I only use PySide and don't mix it with PyQt4, then everything works fine (#481 also fixed, no crash at exit), at least on Windows. Next step is to test this in EPD on Windows with matplotlib, and then on Linux...

@cboos cboos inputhookqt4: make hook more robust in case of unexpected conditions
 - return early if there's no longer a qApp
 - trap any exception that could be raised from within the hook,
   report it and unregister the hook
IPython member

Good find, but I am sure that I am not importing both PyQt and PySide, because I don't have both on this machine. Base on the MPL discussion, it seems like this might be a PySide bug, but there's clearly a way to avoid it, because it doesn't affect IPython master, it only comes up using this PR.


Btw, I noticed since that it was already possible to disable the event loop, by just typing %gui alone.

I see however two problems with that:

  • it's indeed documented (%gui?) but that documentation is in IPython/core/, so I didn't immediately figured out... and being that far away has also the consequence that the documentation is a bit outdated (no mention of the recent GUIs).
  • I would have expected %gui to give me a hint about the currently active GUI, not disabling it! That would also be more consistent in terms of type of return value.

Ok, which version of matplotlib were you using?

IPython member

git master


Ok, I can now also reproduce the above crash on Windows, with matplotlib 1.2.x (2bb91ce).

But even with ipython master (cb6da34) I get a similar crash at exit:

C:\Workspace\src\git\ipython>C:\Dev\Python27-EPD-7.1\python.exe --pylab qt
Python 2.7.2 |EPD_free 7.1-2 (32-bit)| (default, Jul  3 2011, 15:13:59) [MSC v.1500 32 bit (Intel)]
Type "copyright", "credits" or "license" for more information.

IPython -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Welcome to pylab, a matplotlib-based Python environment [backend: Qt4Agg].
For more information, type 'help(pylab)'.
IPython\lib\ RuntimeWarning: PyReadline's inputhook can conflict with Qt, causing delays
            in interactive input. If you do see this issue, we recommend using another GUI
            toolkit if you can, or disable readline with the configuration option
            'TerminalInteractiveShell.readline_use=False', specified in a config file or
            at the command-line

In [1]: %gui qt
Out[1]: <PySide.QtGui.QApplication at 0x46cfcb0>

In [2]: qApp = _

In [3]: x = randn(10000)

In [4]: hist(x, 100)
 <a list of 100 Patch objects>)

In [5]: while True: qApp.processEvents()
KeyboardInterrupt                         Traceback (most recent call last)
C:\Workspace\src\git\ipython\<ipython-input-5-d0d3849ef433> in <module>()
----> 1 while True: qApp.processEvents()


In [6]:
Do you really want to exit ([y]/n)?
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "C:\Dev\Python27-EPD-7.1\lib\", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "c:\workspace\src\git\matplotlib\lib\matplotlib\", line 82, in destroy_all
  File "c:\workspace\src\git\matplotlib\lib\matplotlib\backends\", line 388, in destroy
    self._widgetclosed )
RuntimeError: Internal C++ object (PySide.QtGui.QMainWindow) already deleted.
Error in sys.exitfunc:
Traceback (most recent call last):
  File "C:\Dev\Python27-EPD-7.1\lib\", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "c:\workspace\src\git\matplotlib\lib\matplotlib\", line 82, in destroy_all
  File "c:\workspace\src\git\matplotlib\lib\matplotlib\backends\", line 388, in destroy
    self._widgetclosed )
RuntimeError: Internal C++ object (PySide.QtGui.QMainWindow) already deleted.

Besides, with ipython master there's no event loop integration with PySide. You can however do while True: qApp.processEvents() on the command line to check that matplotlib 1.2.x is indeed working fine with PySide (well, until exit that is ;-) ).

I think that having the crash or not depends on the order in which the atexit handlers are run. I use a VisualStudio 2008 build, and perhaps you have a mingw one? In any case, relying on the order of atexit handlers is fragile at best. I see that matplotlib in tries to detect a destroy() of the FigureManagerQT.window (the one involved in the crash). But this doesn't work, the signal is not delivered to the _widgetclosed method, quite certainly because the objects involved in signal propagation from C++ to Python objects are already gone at that point. See in particular destroyQCoreApplication, the PySide atexit handler which starts by doing a SignalManager::instance().clear(), then proceeds with a bm.visitAllPyObjects(&destructionVisitor, &data) before finishing with a delete app... I wouldn't dare using anything from the PySide API past that point!

Bottom line: it's a matplotlib issue due to PySide behaving differently than PyQt4, nothing really caused by the changes here. Quite the opposite, PySide is better supported as you get event loop integration and with my last commit da13a62, I handle this crash in a more robust way by unregistering the hook. I get a full crash report however, instead of only the backtrace as above, for some reason.

Suggested fix for matplotlib:

diff --git a/lib/matplotlib/backends/ b/lib/matplotlib/backends/
index c85b4b3..64a737e 100644
--- a/lib/matplotlib/backends/
+++ b/lib/matplotlib/backends/
@@ -382,6 +382,7 @@ class FigureManagerQT( FigureManagerBase ):

     def destroy( self, *args ):
+        if QtGui.QApplication.instance() is None: return
         if self.window._destroying: return
         self.window._destroying = True
         QtCore.QObject.disconnect( self.window, QtCore.SIGNAL( 'destroyed()' ),
IPython member

Wow, I don't know how I managed that bad report. You are right, master doesn't work at all in PySide.

And I do know the reason that it's a crash in your branch, and a traceback in master. I fixed that bug last week: d5548fa

If you rebase on master (I just did, it went cleanly), you will just get a traceback. I'm fine with this, then, as it is indeed a matplotlib and/or PySide bug, and doesn't crash IPython. I recomend sending your fix upstream to matplotlib, as it seems clean.


I've been able to test this branch with PyQt, PySide, Tk, Wx, Gtk on Windows and Linux, and also with pylab on both platforms. The test suite passes on both platforms as well. mrink tested pylab with pyside for the Mac. Now all looks good i.e. #481 is fixed for PyQt4 and PySide, no ill side-effects...

Is there anything else I should do? I wonder if I should rework the patchset once more in order to have fewer and only "clean" changesets inside. And one thing I'm unsure about is the impact on py3k support... I don't know exactly what the current status is supposed to be, but I couldn't get master to work with Python 3.2.2.

Of course there are still a lot of improvements that could be made in the IPython/lib/inputhook* area, especially now that I've tested more GUI backends and I see that they could also benefit from this new approach. But I'll save this for another PR (continuing on

IPython member

Python 3 support should be working in master. What problems are you seeing? I'll grab a fresh checkout and have a look.

IPython member

Great, let's wait for @takluyver's report on py3 before we make any final decisions on this one. But it's looking very close to ready for merge, thanks @cboos for the work and patience!

IPython member

Master seems to be working fine in Python 3.2.2 (give or take #939, which I've just opened, but that's a minor problem). I don't think I've experienced the problem this PR aims to solve, so I'm not sure if I can usefully test it.

IPython member

@takluyver, testing this amounts to going to docs/examples/lib and running each of the scripts after starting ipython with --gui X. In each case, you should be able to see a window (possibly just a static one, possibly something with a button or two depending on the example) and retain non-blocking, typing control of the IPython session.

Since I don't know whether we've actually tested this or not with Py3 before, I suggest running those tests first in master and then with @cboos PR merged in. We should only bother this PR if there's a difference after merging it; any problems that may exist in master we'll ignore for the purposes of this PR an deal with separately.

IPython member

Qt and Tk appear to be working fine, both in master and this branch. Gtk and Wx aren't on Python 3 - in fact, Gtk is, but I think our integration is for PyGTK, not the new gobject bindings.

IPython member

OK, I've tested this as much as I can interactively, and given that the rest of us here have also run it through its paces, I'll merge it now. There may be lurking issues (the GUI integration stuff is devilishly hard) but we won't find those until we put it to broader, real-world usage.

@cboos, many thanks for this! I know it was hard work with a lot of back and forth, so I'm glad you stuck with it. It's a necessary hurdle to keep the code quality as high as we can given the small size of our team.

@fperez fperez merged commit 1f7e9d7 into ipython:master Oct 28, 2011

@all: thanks for testing and merging!

@takluyver: regarding py3k, python fails right away with execfile, but even if I can go a bit further by adding an from IPython.utils.py3compat import execfile, it fails a bit later. I see import __builtin__ everywhere, plus some places having Python 2-only syntax (at least in IPython/config/, IPython/utils/ Did I miss something obvious (like I should be on some branch instead of master) or should I create a ticket where we could elaborate?

IPython member

@cboos: You have to do python install, rather than running via, because 2to3 runs during the build process. I use a virtualenv for development.


@takluyver: Ok, got it! Thanks for pointing me in the right direction. It's not yet fully working (compat issues with pyreadline 2.0), but at least now it make sense!

@janschulz janschulz referenced this pull request in matplotlib/matplotlib Oct 4, 2015

plt.plot(...) makes python repl in cmd slow #5159

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment