Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

IPython terminal issue with Qt4Agg on XP SP3 #481

Closed
klonuo opened this Issue · 64 comments
@klonuo

If matplotlib backend in 'matplotlibrc' is set to Qt4Agg, terminal console mode slows down on multiple levels: there is pause on every letter entered; if I paste some code I have to wait approx 1sec for every 2-3 characters!; and overall it seems so unnaturally slow. This does not happen however in IPython qtconsole, and I don't remember similar behavior when I used 10.2 on Windows 7. This also does not happen if matplotlib backend is set to WXAgg for example.

Noticed on:

Windows XP SP3 32 bit
Python 2.6.6 (official from python.org)
PyQt4 4.8.4
matplotlib 1.0.1
IPython 0.11.dev

@fperez
Owner

I'm marking this as critical for 0.11 so that at least we check (I can do it) if one of us can reproduce it. If this is systemic, it will make 0.11 unusable on Windows with Qt.

@fperez fperez was assigned
@jstenar
Collaborator

I can reproduce this problem on windows 7 64-bit.
python2.6.6 (32bit)
pyqt4 v4.8.3
matplotlib 1.0.1
ipython 0.11.dev master 1144388

However it is not a problem in the qtconsole

It does not behave like this with the tk-backend

/Jörgen

@fperez
Owner

@epatters, this looks pretty nasty. Do you have any inkling as to what could be going on? I had the impression you'd been testing with Windows a while ago, so I imagined you'd have seen this already.

It definitely is not a problem in Linux, so somehow it must be something with Qt on Windows. I wonder if it's a Qt 4.8 problem, I think when we did the original work we were testing (I also did some light Windows testing at the time) with Qt 4.6 or 4.7...

@fperez
Owner

@epatters, btw, I just realized my previous message makes it sound like I'm blaming you, which is not the case :) Just looking for your input/wisdom...

@epatters

I've been using PySide on Windows, which means that I can't use matplotlib's Qt backend. In other words, I haven't tried this out lately. When I have access to my Windows box tomorrow, I will switch back to PyQt4 and see if I can debug the problem.

Off the top of my head, though, the behavior described above also occurs when running a PDB session under vanilla Python if QtCore.pyqtRemoveInputHook() has not been called. For more information, see:

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/python_shell.html

Perhaps something is not playing nice with PyQt4's input hook...

@jstenar
Collaborator
@fperez
Owner
@klonuo

Feel free to contact me with instructions if you don't have Windows XP/7 PC. I have both with IPython 0.11dev

Cheers

@klonuo klonuo closed this
@klonuo

Well, I have no idea how I closed this issue as I only commented

huh

@epatters

The "Comment and Close" button is dangerously close to the "Comment" button. Not a great usability decision there...

@epatters epatters reopened this
@minrk
Owner

That's true. I use dotjs (Userscripts/greasemonkey would work too) to float it to the left and fade it a little, so I don't click it accidentally:

function leftClose(){
    // float 'Comment & Close' button on the left, to prevent accidental clicking
    // also fade text to grey to make it stand out less
    // query from from userscripts.org/scripts/105325
    var andClose = document.querySelector('button[name="comment_and_close"]');
    if (andClose){
        // float it on the left
        andClose.style.float='left';
        // fade text
        andClose.style.color="#aaa";
        // or you can just remove the button altogether:
        // andClose.parentNode.removeChild(andClose);
    }
}
leftClose();

This is what it looks like to me:

@epatters

Nice one! Thanks.

@klonuo

I noticed same behaviour with visvis. In this example:

In [1]: import visvis as vv
In [2]: app = vv.use()
In [3]: f = vv.clf()
In [4]: a = vv.cla()

after input [3] IPython console slows down multiple times.

Obviously regardless matplotlibrc backend settings

@klonuo

Just noticed that visvis default backend is set to Qt4, and when set to other no slowdowns
So this seems like 'expected behavior' then

@epatters

I confirmed this problem in my Windows 7 VM. I should add that I also experience this on my OS X machine, and though the delay is much less noticeable, I do not think I am imagining it.

Notice this comment from the PyQt4 docs that I linked above: "The installation of an input hook can cause problems for certain applications (particularly those that implement a similar feature using different means)."

I have no idea how to go about making these input hooks play nicely together.

@epatters

Actually, further testing shows that the delay is just as bad on OS X as Windows. As it stands now, the interaction between pyreadline and PyQt4's input hook makes the terminal IPython effectively useless when using PyQt4 (this includes the Qt pylab backend).

@takluyver
Owner

Is there anything we can do about this before 0.11? Do we need to put a warning in the docs (or in the code)?

@minrk
Owner

That's pretty rough, though at least Windows is the only place that pyreadline is normally used. I don't see any issues when using real readline on OSX.

@epatters

Hmm. If I run ipython pylab=qt on OS X and then plot something, I experience the delays.

@klonuo
@minrk
Owner

Weird, I can't cause any problems at all.

What's your Python, matplotlib, Qt, etc. situation?

@epatters

My configuration is:

Python - 2.7.1
IPython - master branch
matplotlib - 1.0.1
Qt - 4.7.3
PyQt - 4.8.2

@epatters

OK, this is even weirder than I thought. Only sometimes do I experience the 2-3 second delays. Sometimes it appears to work fine.

I am thoroughly baffled.

@minrk
Owner

The only major difference I have is I'm using system Python 2.6, but I wouldn't expect that to make a difference. I don 't particularly want to go through the bother of building pyqt for 2.7, but maybe I have to.

I've tried both matplotlib 1.0.1 and git master with no change.
Do you have readline, as installed by 'easy_install readline', or something else?

@fperez
Owner

@epatters, just for my understanding: you're seeing the slowdown happening with normal readline, right? Pyreadline is win32-only, if I remember correctly...

@epatters

I have a readline module:

In [4]: import readline

In [5]: readline
Out[5]: module 'IPython.utils.rlineimpl' from '/Users/epatters/Local/IPython/IPython/utils/rlineimpl.pyc'

I do not have pyreadline installed.

@minrk
Owner

That will be reported no matter what readline you have. You need to check readline._rl to see the real module if you are in IPython.

@fperez, yes pyreadline is win32 only, but I could swear that I have installed it accidentally on OSX ages ago when I didn't know what name to easy_install.

@minrk
Owner

As for what to do on the matter, since IPython requires pyreadline on Windows, we essentially have to say "don't use the matplotlib Qt backend with IPython until this gets worked out".

We should put a warning describing the issue in the Qt GUI activation if pyreadline is detected.

@fperez
Owner

Thanks, @minrk, that was the source of my slightly confused question. This looks really mysterious. I'm quite stumped too...

@fperez
Owner

@minrk, your plan for win32 sounds like the right thing to do, though I'd go as far as adding special code in there that spits out a very loud warning at runtime if people try to use the win32/pyreadline/qt combo, so the normal users who don't read docs still understand what's going on.

As for osx, total mystery...

@minrk
Owner

@fperez by 'warning' I did mean actual warnings.warn inside inputhook.activate_qt4, not a comment for developers.

We will have to investigate further on osx, but I can't seem to reproduce it.

@fperez
Owner

Ah, OK, thanks. That's what I meant too, just misunderstood you.

@jkmacc

I'm experiencing the same problems on a Mac. Any invocation of Qt gui elements severely hangs/lags the terminal and spikes CPU usage. This behavior is inconsistent, though, usually but not always happening, and regardless if which ipython threading flags I supply (it also happens in the regular python prompt). My matplotlib backend is 'TkAgg'.

OS X 10.6.7
Python - 2.6.1 (Apple-supplied)
IPython - master branch
matplotlib - 1.0.0
PyQt - 4.8.4
Qt - 4.7

@fperez
Owner

I'm removing this from being an 0.11 blocker, since for now all we can do is issue the warning that's been put in. Hopefully we'll understand better what the problem is later on.

@cboos

Hello,

I've installed ipython-0.11 and pyreadline-1.7 for Python 2.7.2 on Windows 7, and I also have this issue, with PyQt-4.8.5-1.

Looking into the details, I see that pyreadline's Console.get calls the PyOS_InputHook and that this gets called many times per character typed (key pressed, released). In our case that hook is the not-so-cheap qtcore_input_hook, the one which has been reinstalled by the call to pyqtRestoreInputHook in enable_qt4. That hook basically does the following (translated in Python, the original C++ version is in sip/QtCore/qcoreapplication.sip):

timer = QTimer()
timer.timeout.connect(app.quit)
while True:
    timer.start(100)
    app.exec_()
    timer.stop()
    if kbhit():
        break

Two problems here: there will be always at least 100ms spent for each call, even if there's a pending key event.
So instead of this, I've replaced it with a custom hook which has a shorter timeout and more importantly doesn't exec_ if there's a pending event.

This seems to fix the problem for me. I've only tested this on Windows, but I think it will eventually help for Mac as well and hopefully won't break other stuff... should I create a pull request with the patches nevertheless or do you prefer an other form of communication for "experimental" changes?

@fperez
Owner

@cboos, pull request is good, just in it, link back to this discussion and mention that it's not intended for immediate merge but rather to explore the problem. PRs are still a nice way for everyone to be able to pull the changes and to centralize the discussion about the code so go ahead with one.

And thanks! It will be great to make headway on this nasty one...

@cboos

See #815 for the proposed fix.

@cboos cboos referenced this issue from a commit in cboos/ipython
@cboos cboos inputhook: move the new CTRL+C handling logic to this level
The inputhookqt4 module now only contains a simple, straightforward
inputhook_qt4 input hook doing only Qt4 event processing in short
batches (#481).

The logic to handle CTRL+C is now entirely done at the level of the
InputHookManager, where a new set_safe_inputhook() method can be used
to install a hook which will automatically benefit of the two-step
CTRL+C handling (first for interrupting the event loop and suspending
the input hook, second for returning to the prompt, at which point the
input hook is restored). This should help to address issues #122 and
catch KeyboardInterrupt and has to be installed using
set_safe_inputhook).

Some extra care is taken on Unix where we want to make sure no
KeyboardInterrupt is raised upon entering the Python callback (e.g. in
readline.c, if a KeyboardInterrupt is received during a select(), the
PyOS_InputHook is executed directly afterward and if it's a ctypes
Python callback the exception will be raised immediately on entering
the function, even before an initial `try:` could be executed).
1d6e9a0
@fperez fperez closed this in 1f7e9d7
@fperez
Owner

Note: I just merged #815 that closed this automatically, but since these gui/event loop problems have a habit of reappearing despite lots of testing, if anyone notices anything amiss here, please do reopen. There's always a chance that #815 wasn't sufficient for some bizarre combination of python/os/gui/phase of the moon...

@cboos

Note in particular that these symptoms will reappear if anything calls QtCore.pyqtRestoreInputHook()!

@takluyver
Owner

Does it make sense to monkeypatch that so it's a no-op while our own inputhook is in place?

@fperez
Owner

@takluyver, that sounds like a sensible idea to me; @epatters, do you have any thoughts on the matter?

@epatters

I think it's alright to monkey-patch it. The only use case I know of for pyqtRemoveInputHook and pyqtRestoreInputHook is break-point debugging: you sandwich calls to pdb.set_trace between these guys to prevent complaints from Qt about the event loop. In this case, ignoring the request to restore the input hook is the right thing to do, as it wasn't there to begin with.

@fperez
Owner

Thanks for the feedback, @epatters. In that case, anyone who wants to whip up a quick monkeypatch for that particular function will make us happy :) Hopefully that will bury this nasty issue once and for all; it took quite a while!

@SirVer

I'd like to reopen this issue (but I do not have permission to do so) because I have trouble with gui qt in a very recent clone (65ac74e; yesterday evening). I noticed slugginesh behavior on the ipython shell on two Mac OS X 10.6 Systems and did some experiments.

I used the following unscientific benchmark: I copied and quickly pasted a long text (4500 chars, all one line) in two shells - both ran ipython default profile but in one of them I ran a %gui command before pasting. First experiment was with %gui qt and I noticed that the other ipython shell always finished ~.5 seconds before the gui ipython - even if i pasted in the gui shell first. I then repeated the experiment using %gui osx and the gui shell was as quick as the non gui shell.

The slugginess really affects working especially when walking through the history: I very often press a twice because the reaction comes notably late to the shell.

I noticed that replacing timer.start(50) in lib/inputhookqt4.py:91 with timer.start(1) fixed the problem for me - but upped CPU usage from 30%. Also, I really doubt that I would notice a reaction time of 50ms though - I rather think that there is more delay than just those 50ms. Maybe some with more qt experience could help out. I find this code really hard to get - starting and quitting the app periodically doesn't seem like a canonical way of doing things and imho could have implementation detail dependent side-effects on some OSes.

Another fix that works for me like a charm without breaking any functionality (tested using --profile=pylab in this case) is to completely kill the whole block starting in line 88: if not stdin_ready():
In my understanding, my QT windows should no longer react now because the input hook is never called when I do no type inside the ipython shell. But that is not the case, the windows react and there is no felt sluggines in the shell anymore; also the CPU usage remains < 1 %. If I print something in the inputhook, I see that it is called ~10-20 times per second, so it seems good enough to leave the QTApp some 300ms to work its events every time the inputhook is called. I therefore suggest this as fix: SirVer/ipython@8af05fa

Below are the versions of one of the boxes (I do not have the other computer around atm):

ipython github 65ac74e
OS X 10.6.8
Python 2.7.1 (build with homebrew)
matplotlib - 1.0.1
PyQt - 4.8.3
Qt - 4.7.1

@fperez fperez reopened this
@fperez
Owner

@SirVer, I've just reopened it. Thanks for all the detailed info, we should be able to make some progress with this...

@cboos

I just tested 8af05fa, but as I expected this kills the interactivity on Windows as the hook is not called when the process is idle on that platform (well, I must admit I have no idea what happens on a Mac and can't investigate either as I don't have access to one).

@cboos

... but the interactivity is indeed preserved on Linux! This is due to raw_input, which calls PyOS_Readline / call_readline, and finally readline_until_enter_or_signal which calls PyOS_InputHook in a loop, temporizing every 0.1s (well, this is probably old news for the pyreadline authors around ;-) ).

If I disable the readline extension, then the behavior seen is the same as on Windows (the change kills the interactivity with Qt).

Also, with only that change a CTRL+C is now simply ignored, not sure why at this point.

@SirVer

I have to add that my 'patch' will indeed reduce the respeonsiveness of (i.e.) matplotlib plot windows - so it is not golden after all but still significantly better than having a lagging shell. I will try out every suggestion on mac os x that is provided - I feel like I cannot provide many suggestions due to lack of knowledge about readline and qt

@fperez
Owner

I don't have really much to offer, other than to thank you guys for pounding on this. It has proven to be a very, very stubborn and hard to fix bug and has most everyone stumped. So even partial progress on it would be very welcome.

@cboos

So to summarize that part of the problem:

  • when readline is used on non-windows platform, the inputhook is called repeatedly by readline (which gives the impression that a simple qApp.processEvents() in the inputhook is enough for guaranteeing interactivity) but readline also wait for 100ms when there's no keypress (which must be what reduces the responsiveness in matplotlib widgets)
  • OTOH, if only the inputhook is in control, the current version of the code doesn't immediately exit when a character is pressed so there's a 100ms delay for detecting the first character; the remaining readable characters are taken into account immediately but in case of copy/paste of a long text I can imagine this get chunked and that we see many times the 100ms delay

I think the solution is simply to do what the original PyQt qt_input_hook did in the non-Windows case, add a socket notifier on stdin for quitting the qApp. I'll take on a patch shortly...

@minrk
Owner

@SirVer - can you reproduce the problematic behavior with current (1.1.0) matplotlib, or better yet, without matplotlib at all, just using --gui qt?

@cboos

Note that I've reproduce the problem as well with just --gui qt, but my idea to add a socket notifier didn't help, I got the same extra delay. I think that what might help is to revert to using the original pyqtRestoreInputHook approach in this situation (non-windows + readline + PyQt). See 7f643b3 for details.

But there's worse: while testing this on Windows with --gui qt, it seems one can "freeze" ipython when pasting a moderately sized text (1561 characters if that rings a bell for someone; no idea myself yet). Disabling pyreadline makes no differences, we have a freeze in either case. Without --gui qt, I can paste a 10k text without trouble (sloooow though with pyreadline, instantaneous without). It seems that this bug predates my changes though, as I have the exact same effect on a3229f5 (the parent cset of my patch series).

Also, note that this "freeze" only concerns the console input, Qt windows remain responsive.

@rkern

For what it's worth, an EPD customer has seen similar problems on Windows XP with the WXAgg backend to matplotlib. Changing the inputhook constants do not seem to make any noticeable difference. I have not been able to replicate the wx slowness myself on Windows 7 (I can replicate the Qt slowness, though).

@SirVer

sorry for the late feedback, I was travelling and didn't bother doing my email. Yes, I can also see the problem just with using --gui qt.

@fperez fperez referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
@Proletariat99

Here is a VERY simple newbie solution for someone brand new to python/iPython that may work for some of you newbies like me:

Here's my version info:

  • Python 2.7.2 -- EPD_free 7.2-2 (32-bit)
  • ipython --version 0.12
  • matplotlib 1.1.0
  • numpy 1.7.0
  • scipy 0.11.0 maybe?

I just created a new profile using the startup option '' and that seemed to do the trick. Utterly newb solution, but it worked for me and may work for you.

For you seasoned devs, here's some info that may help:

Before I found the hidden nanolink at Enthought, I struggled with the install for quite a while. After attempting to install manually a variety of ways, I think I had a very confused python install. I'm guessing that I had something in there that called a problematic input hook that was causing the problem. I wasn't able to find anything, but I honestly didn't look that hard yet.

Here are just a few of the misguided things I did while trying to install:

  • gcc manual install (was trying to avoid xcode - failed - no compiler)
  • clang install (failed)
    • llvm manual install (clang dependency)
      • freetype2 manual install (llvm dependency)
  • Finally gave in to xcode install (hate xcode -- big, closed, clunky)
  • Heard "Enthought" enough to realize they HAD to have a free link somewhere that had Numpy/Scipy/Matplotlib as a free package. Went back and scoured their site until I found it.

So then I finally created a new profile and it fixed the super slowness.

Hope that helps someone, somewhere.

Cheers.

@takluyver
Owner
@fperez
Owner

I'm sorry to do this, but I'm downgrading this to 'high' priority from 'blocker', simply because we can't realistically block the entire release on an issue we have no clue how to fix.

It's a real bummer, and I hope we'll be able to track what's going on at some point, but absent any progress, we can't stall a release on this. And in fact, we were violating the 'blocker' marker already since we'd bumped this from 0.11 to 0.12 to 0.13, so if anything I'm just being upfront about what we've already done.

@indiajoe

I would like to report that i have been facing this problem for a long time in my gnu/linux too. Surprised to see it has been reported only in Windows.


In short following are the specifications:
Linux Kernel : 3.5.2
Distro: Porteus ( https://en.wikipedia.org/wiki/Porteus_%28operating_system%29 )
Desktop: Trinity (aka. fork of KDE 3.5 )
Python version : 2.6.4
IPython version : 0.12
I run Ipython in terminal by the simple "ipython" command. (no extra arguments)


Description of problem:
This usually happens when i leave Ipython terminal open for a few days.
I typically have the numpy, scipy and matplotlib modules loaded.
The machine doesnot gets slow. Infact the commands run by Ipython are also not slow.
Only the keyboard input is slow.
Even if I copy-paste some content into Ipython, it prints each pasted letter one by one with a uniform delay.
While this pasted content is being printed slowly on screen or when i use backspace to remove it, we can see the CPU percentage use of Ipython shooting significantly up in htop.
If I exit the session, and start again, everything works fine as usual.


@oliford

I and my office mate both see this bug in linux too. So here's our experiences, in case it helps isolate it:

I'm using XUbuntu 12.04 (Xfce), ipython 0.12.1 and my office mate also on Linux Mint with ipython 13.1-rc2.
We both see it slowing only the keyboard input (including just moving the cursor) and pasting after use of matplotlib and numpy for some time. I do also use scipy but my officemate doesn't. We're both using the GTKAgg backend.

It doesn't occur immediately, but after some time with use of plotting etc.
We both see significant disk activity during moving the cursor back and forth (as in #1184, perhaps I should have posted this there? I don't want to double post though).

@oliford

I've found that our problem is exactly that in matplotlib/matplotlib#1530 relating to writing .local/share/recently-used.xbel after opening the GTK (maybe also Qt) save figure dialog once (but not necessarily saving anything). I'll wander over there now.

@danielballan

I have the same experience as @oliford on Ubuntu 12.04, iPython 0.14.dev, GTKAgg.

@danielballan

One man's testimonial: I resolved this issue by installing the dev version of matplotlib. Perhaps this commit in this issue cited by @oliford did the trick.

@minrk
Owner

Based on @danielballan's feedback, I would be inclined to close.

@ivanov
Owner

closing based on the conversation around matplotlib/matplotlib#1530.

@ivanov ivanov closed this
@damianavila damianavila referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.