Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Ensure %gui qt works with new Mayavi and pylab #8

Closed
ipython opened this Issue May 10, 2010 · 25 comments

Comments

Projects
None yet
7 participants
@ghost

ghost commented May 10, 2010

Original Launchpad bug 508953: https://bugs.launchpad.net/ipython/+bug/508953
Reported by: fdo.perez (Fernando Perez).

Right now mayavi/qt is only on SVN, but eventually we need to ensure that %gui qt allows mayavi/qt and pylab/qt to work together.

For more details:
http://mail.scipy.org/pipermail/ipython-dev/2010-January/005728.html

@ghost

ghost commented May 10, 2010

[ LP comment 1 by: Michele Mattioni, on 2010-01-18 11:54:43.261168+00:00 ]

Expected:

  • The window should respond to the command (zoom, rotate...)
    and not freeze.
  • The prompt should be back and be responsive.
@ghost

ghost commented May 10, 2010

[ LP comment 2 by: Michele Mattioni, on 2010-01-18 12:02:03.250363+00:00 ]

Ok, this is weird.

Sometimes the window freeze, sometimes respond to the command (like zoom and rotate).

However I can't get the prompt back and ready to insert a new command.

@ghost

ghost commented May 10, 2010

[ LP comment 3 by: Brian Granger, on 2010-01-18 16:42:52+00:00 ]

Mayavi and pylab are probably doing some funny things with the event
loop that is messing up
things. It will probably require someone looking through the
Mayavi/pylab/traits code carefully
to look for these things.

Cheers,

Brian

On Mon, Jan 18, 2010 at 4:02 AM, Michele Mattioni mattions@gmail.com wrote:

Ok, this is weird.

Sometimes the window freeze, sometimes respond to the command (like zoom
and rotate).

However I can't get the prompt back and ready to insert a new command.

Ensure %gui qt works with new Mayavi and pylab
https://bugs.launchpad.net/bugs/508953
You received this bug notification because you are a member of IPython
Developers, which is subscribed to IPython.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:
Right now mayavi/qt is only on SVN, but eventually we need to ensure that %gui qt allows mayavi/qt and pylab/qt to work together.

For more details:
http://mail.scipy.org/pipermail/ipython-dev/2010-January/005728.html

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger@calpoly.edu
ellisonbg@gmail.com

@ghost

ghost commented May 10, 2010

[ LP comment 4 by: Fernando Perez, on 2010-01-18 23:16:28.353993+00:00 ]

The attached script can be run in ipython trunk-dev, and shows the problem:

maqroll[Desktop]> ip
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
Type "copyright", "credits" or "license" for more information.

IPython 0.11.bzr.r1346 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: gui -a qt
Out[1]: <PyQt4.QtGui.QApplication object at 0x93086ac>

In [2]: run qt_embedding.py

Button pushed

Terminal is frozen here, until we close the mayavi window.

In contrast, if we run the same script in 0.10 with -q4thread:

(ipython-0.10)maqroll[Desktop]> ip -q4thread
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
Type "copyright", "credits" or "license" for more information.

IPython 0.10 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.

In [1]: run qt_embedding.py

In [2]: 1+1 # terminal accepts input though window remains open
Out[2]: 2

In [3]: Button pushed

Thanks to Gael for the feedback to fix the api calls in the example script.

Owner

ellisonbg commented May 13, 2010

I think this is related to the larger issues we are having with Traits and IPython's new even t loop stuff.

Owner

fperez commented Mar 23, 2011

The example script is at (since we can't attach anything on gh yet):

https://bugs.launchpad.net/ipython/+bug/508953/+attachment/1107754/+files/qt_embedding.py

That script still blocks, but I suspect that it's a matter of updating the script to use the event loop integration code we now have. I'm going to put Brian and myself on this one so we sort it out, because what we probably need is simply to finish documenting how this must be done.

Normal use of mayavi via mlab functions now works fine with Qt, including multiple figure, matplotlib integration, etc.

Owner

fperez commented Mar 23, 2011

So in the end, this may be either bug or docs, but I want to discuss it with Brian before making the final call.

@ghost ghost assigned fperez, ellisonbg and fperez Apr 10, 2011

Owner

fperez commented Apr 10, 2011

@ellisonbg, I'll take care of this one. It's pretty clear that for 0.11 we'll have traits/mayavi/mpl support but not (and perhaps never) generic gui support. So I think code-wise we're done, and what's left to be done is documentation.

i'm looking forward to this piece of documentation - as i would like to understand how a non-blocking ipython behavior with a pyglet window(s) could be achieved.

Owner

fperez commented Apr 12, 2011

I won't have time to update the docs right away, but you may want to look at this file:

https://github.com/ipython/ipython/blob/master/IPython/lib/guisupport.py

The basic idea is to initialize any singleton objects and the event loop only once. But the gui event loop will need to be integrated with python's inputhook, here's the integration we have so far:

https://github.com/ipython/ipython/blob/master/IPython/lib/inputhook.py
https://github.com/ipython/ipython/blob/master/IPython/lib/inputhookgtk.py
https://github.com/ipython/ipython/blob/master/IPython/lib/inputhookwx.py

it may be necessary to write similar support for pyglet. We'd welcome a contribution on this front!

Thanks for the links. Strange enough, it already seems to work for our fos project http://fos.me - we use pyglet and some of sympy logic. Running an example script opens the window with the OpenGL context, the Ipython shell stays interactive and it possible to add and remove actors (without calling ipython with -q4thread or -wxthread).

==EDIT==
For fos, we managed to use pyglet's OpenGL bindings through Pyside with the new Ipython 0.11 threading model. In case somebody is interested, the code is here: https://github.com/fos/fos-pyside/

Owner

fperez commented Apr 13, 2011

OK, thanks for letting me know. Please post any additional (good or
bad) info you find out about this, so we can mention it when we update
the docs.

Owner

fperez commented May 29, 2011

Hey @rkern, @epatters, I think I could use some help on this one from you guys... I can get the docs sorted out, all matplotlib support seems fine, and mayavi can also be run, but I'm getting a weird problem that I have no clue what to do with... The following code can be used to test and illustrate, and shows the issue with just traits so we know it has nothing to do with mayavi/vtk/opengl/etc:

"""Tests for Traits GUI behavior

%run simpletraits [backend]

where the optional backend can be one of:  qt4, wx.

This code is crashing with Qt but OK with WX...
"""
from __future__ import print_function

import sys
import os
import time

ip = get_ipython()

try:
    toolkit = sys.argv[1]
except IndexError:
    toolkit = 'qt4'

app = ip.magic('gui %s' % toolkit)

print("using toolkit:", toolkit)
os.environ.pop('ETS_TOOLKIT', None)

from enthought.etsconfig.api import ETSConfig
ETSConfig.toolkit = toolkit

from enthought.traits.api import HasTraits, Float
class X(HasTraits):
    value = Float(0)

x = X()
x.edit_traits()

print("Sleeping for 2 seconds, hit Ctrl-C or wait until prompt returns")
sys.stdout.flush()
time.sleep(2)

With the WX backend, you can run this as many times as you want and it's fine, and you can interrupt the last time.sleep() call (or a manually made one) and it behaves OK (IPython started without any command-line flags at all, ETS_TOOLKIT not set at all):

In [1]: run simpletraits.py wx
using toolkit: wx
Sleeping for 2 seconds, hit Ctrl-C or wait until prompt returns
^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
/home/fperez/ipython/test/simpletraits.py in ()
     36 print("Sleeping for 2 seconds, hit Ctrl-C or wait until prompt returns")
     37 sys.stdout.flush()
---> 38 time.sleep(2)

KeyboardInterrupt: 

In [2]: 

But if we run this against the Qt backend (on my ubuntu 10.10 box, with the ubuntu-provided packages), I get:

In [1]: run simpletraits.py qt4
using toolkit: qt4
Sleeping for 2 seconds, hit Ctrl-C or wait until prompt returns
^C
amirbar[test]>   ### Back to system shell, instant and silent crash

Basically, any Ctrl-C use once the traits machinery has been initialized with the Qt backend will instantly crash the interpreter, no messages, nothing.

So the questions are:

  • can you guys reproduce this on your ends?
  • if so, do you have any idea of what could be going on? Would the fix be on our side or the Traits/ETS side?

I hadn't seen this failure mode before, and it's obviously pretty serious. I know many traits users are moving to Qt quickly, so putting out an ipython that segfaults on Ctrl-C with any traits code would be not very nice.

I've tried installing signal handlers myself to no avail, and the code won't produce a core dump. I tried stracing the process, and all I get is this:

....
select(0, NULL, NULL, NULL, {5, 0}

)     = ? ERESTARTNOHAND (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
Process 11303 detached

where I had hit Ctrl-C while time.sleep(5) was waiting.

I'm really quite stumped, so help from any front would be great. I'll ping the list as well.

As a final data point, Ctrl-C behaves OK with matplotlib using the Qt4 backend, the problem only shows up once any traits code is activatedm as illustrated here:

amirbar[test]> ip -pylab qt4 --no-banner

Welcome to pylab, a matplotlib-based Python environment [backend: Qt4Agg].
For more information, type 'help(pylab)'.

In [1]: plot(rand(100))
Out[1]: []

In [2]: import time;time.sleep(5)
^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
/home/fperez/ipython/test/ in ()
----> 1 import time;time.sleep(5)

KeyboardInterrupt: 

In [3]: from enthought.mayavi import mlab

In [4]:  # Ctrl-C here
amirbar[test]>   # instant crash
Owner

fperez commented May 29, 2011

I should correct something in the above: we're not really seeing a segfault, rather an uncaught SIGINT that simply interrupts the interpreter. The problem is basically that for some reason, SIGINT is being handled by Qt, ignoring any handler we may try to install at the python level with signal.signal. But from the perspective of the user it feels like a crash, since there's an immediate, unconditional termination of the work session.

Contributor

epatters commented May 30, 2011

I know what the culprits are here:

https://github.com/enthought/traitsui/blob/master/traitsui/qt4/toolkit.py#L36
https://github.com/enthought/pyface/blob/master/pyface/ui/qt4/gui.py#L105

These were added for the reason described in the comment: sometimes KeyboardInterrupts would fail to interrupt the application because Qt ate them. Obviously, this is not an acceptable solution for interactive use. It is probably best to just remove them. If Ctrl+C doesn't always kill Qt applications, then so be it. At least it won't lead to surprising behavior like this.

@rkern
What do you think?

Contributor

rkern commented May 30, 2011

+1. It might be worth adding a GUI.allow_sigint() classmethod, or similar, so that we can explicitly enable that behavior in our standalone GUI applications.

Contributor

epatters commented May 30, 2011

Should be fixed in:

enthought/pyface@831f6f3
enthought/traitsui@391f9ac

This may be worth a port to the old-namespace branch.

Owner

minrk commented Jun 27, 2011

@epatters Does this mean this issue should be closed as far as IPython is concerned?

Contributor

epatters commented Jun 27, 2011

I think so. That said, I will test it out this week and then close it.

Contributor

epatters commented Jun 28, 2011

I confirmed that the quitting issue is fixed, but another problem remains with PyQt4: incompatible SIP APIs. ETS (like IPython) uses the v2 QString and QVariant APIs, which are the default under Python 3 PyQt4 and the only option under PySide. Unfortunately, the default is v1 for PyQt4 under Python 2, so when the IPython sets the Qt4 toolkit here:

https://github.com/ipython/ipython/blob/master/IPython/lib/inputhook.py#L184

the SIP APIs become fixed in stone, and subsequent ETS imports fail. One workaround to make sure that ETS imports are done before selecting the GUI toolkit in IPython. Another is to import from IPython.external.qt instead of PyQt4 in the IPython kernel. This would have the following benefits:

  1. The IPython kernel would support PySide by the same mechanism as the IPython Qt frontend and ETS
  2. ETS will always import correctly

The downside is that code relying on the old SIP APIs will break. Matplotlib's trunk version will do the right thing, but 1.0.1 version will not, which is probably unacceptable for the 0.11 release.

In summary, the situation stinks and is bound to create a great deal of confusion.

Owner

minrk commented Jun 28, 2011

That's really ugly, thanks for working it out, though.

I imagine the most common combination for people the least likely to be able to handle doing something weird is Python 2, PyQt4, matplotlib release, so that's probably the most important situation to not break.

If we make sure matplotlib is imported prior to setting the inputhook, then we could have the following logic in the enable_qt4 method in inputhook:

if matplotlib has been imported and is <= 1.0.1: import PyQt4
else: import external.qt

Contributor

epatters commented Jun 28, 2011

From the enable_pylab method in InteractiveShell, it seems that matplotlib will always be imported before the inputhook, so I will implement the hack as you suggest.

Of course, if matplotlib is imported manually or someone tries to use ETS in pylab mode, all bets are off.

Owner

minrk commented Jun 30, 2011

Great! @epatters, how is the patch going? I think this might be the last non-docs code we are waiting for before we cut 0.11.

Contributor

epatters commented Jun 30, 2011

OK, sorry. This is not hard; I've just been busy. I'll have something submitted tonight.

Owner

minrk commented Jun 30, 2011

Sorry, I don't mean to imply there is a rush, if you need the weekend or something.

We do still have a bit of docs to finish.

@epatters epatters pushed a commit to epatters/ipython that referenced this issue Jul 1, 2011

epatters Support v2 PyQt4 APIs and PySide in kernel's GUI support.
Closes gh-8.
cea5ae0

@epatters epatters closed this in 272e128 Jul 2, 2011

@ellisonbg ellisonbg referenced this issue in minrk/ipython May 24, 2013

@ellisonbg ellisonbg Merge pull request #8 from minrk/jsreorg
CSS tweaks
02876cc

@minrk minrk referenced this issue in minrk/ipython Jul 1, 2013

@ivanov ivanov Merge pull request #8 from slojo404/rst2ipynb_update
Rst2ipynb update
27083c8

@minrk minrk referenced this issue in minrk/ipython Aug 28, 2013

@Zsailer Zsailer Merge pull request #8 from ellisonbg/multidir-review1
Review and refactoring of notebooks web service.
842fe02

@minrk minrk added a commit that referenced this issue Oct 21, 2013

@minrk minrk Merge pull request #8 from ivanov/split-below
split focuses the top of the newly created cell
2ced4f6

@takluyver takluyver added a commit that referenced this issue Nov 4, 2013

@takluyver takluyver Merge pull request #8 from ivanov/ipython-dir-default
main change is adding a what's new entry
36cf890

@jasongrout jasongrout referenced this issue in jasongrout/ipython Jan 8, 2014

@jdfreder jdfreder Merge pull request #8 from jasongrout/backbone-flat
options on child views and automatically accumulating _keys from superclasses
1831d13

@mattvonrocketstein mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

@minrk epatters + minrk Support v2 PyQt4 APIs and PySide in kernel's GUI support.
Closes gh-8.
ca6a499

@mattvonrocketstein mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

@ellisonbg ellisonbg Merge pull request #8 from minrk/jsreorg
CSS tweaks
7457926

@mattvonrocketstein mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

@minrk minrk Merge pull request #8 from ivanov/split-below
split focuses the top of the newly created cell
5d06baf

@mattvonrocketstein mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

@takluyver takluyver Merge pull request #8 from ivanov/ipython-dir-default
main change is adding a what's new entry
a9beb5f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment