IPython terminal issue with Qt4Agg on XP SP3

[klonuo]

If matplotlib backend in 'matplotlibrc' is set to Qt4Agg, terminal console mode slows down on multiple levels: there is pause on every letter entered; if I paste some code I have to wait approx 1sec for every 2-3 characters!; and overall it seems so unnaturally slow. This does not happen however in IPython qtconsole, and I don't remember similar behavior when I used 10.2 on Windows 7. This also does not happen if matplotlib backend is set to WXAgg for example.

Noticed on:

Windows XP SP3 32 bit
Python 2.6.6 (official from python.org)
PyQt4 4.8.4
matplotlib 1.0.1
IPython 0.11.dev

[fperez]

I'm marking this as critical for 0.11 so that at least we check (I can do it) if one of us can reproduce it. If this is systemic, it will make 0.11 unusable on Windows with Qt.

[jstenar]

I can reproduce this problem on windows 7 64-bit.
python2.6.6 (32bit)
pyqt4 v4.8.3
matplotlib 1.0.1
ipython 0.11.dev master 1144388

However it is not a problem in the qtconsole

It does not behave like this with the tk-backend

/Jörgen

[fperez]

@epatters, this looks pretty nasty. Do you have any inkling as to what could be going on? I had the impression you'd been testing with Windows a while ago, so I imagined you'd have seen this already.

It definitely is not a problem in Linux, so somehow it must be something with Qt on Windows. I wonder if it's a Qt 4.8 problem, I think when we did the original work we were testing (I also did some light Windows testing at the time) with Qt 4.6 or 4.7...

[fperez]

@epatters, btw, I just realized my previous message makes it sound like I'm blaming you, which is not the case :) Just looking for your input/wisdom...

[epatters]

I've been using PySide on Windows, which means that I can't use matplotlib's Qt backend. In other words, I haven't tried this out lately. When I have access to my Windows box tomorrow, I will switch back to PyQt4 and see if I can debug the problem.

Off the top of my head, though, the behavior described above also occurs when running a PDB session under vanilla Python if QtCore.pyqtRemoveInputHook() has not been called. For more information, see:

http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/python_shell.html

Perhaps something is not playing nice with PyQt4's input hook...

[jstenar]

fperez skrev 2011-06-23 00:25:

@epatters, this looks pretty nasty. Do you have any inkling as to what could be going on? I had the impression you'd been testing with Windows a while ago, so I imagined you'd have seen this already.

It definitely is not a problem in Linux, so somehow it must be something with Qt on Windows. I wonder if it's a Qt 4.8 problem, I think when we did the original work we were testing (I also did some light Windows testing at the time) with Qt 4.6 or 4.7...

I tried to run without pyreadline and then it behaves normally. So there
seems to be some kind of interaction between pyreadline and Qt4.

Unfortunately I'm leaving for a short midsummer holiday and will be back
on sunday evening.

/Jörgen

[fperez]

On Thu, Jun 23, 2011 at 1:30 AM, jstenar
reply@reply.github.com
wrote:

I tried to run without pyreadline and then it behaves normally. So there
seems to be some kind of interaction between pyreadline and Qt4.

Thanks for the feedback though, this is already useful.

Have a good wekend!

f

[klonuo]

Feel free to contact me with instructions if you don't have Windows XP/7 PC. I have both with IPython 0.11dev

Cheers

[klonuo]

Well, I have no idea how I closed this issue as I only commented

huh

[epatters]

The "Comment and Close" button is dangerously close to the "Comment" button. Not a great usability decision there...

[minrk]

That's true. I use dotjs (Userscripts/greasemonkey would work too) to float it to the left and fade it a little, so I don't click it accidentally:

function leftClose(){
    // float 'Comment & Close' button on the left, to prevent accidental clicking
    // also fade text to grey to make it stand out less
    // query from from userscripts.org/scripts/105325
    var andClose = document.querySelector('button[name="comment_and_close"]');
    if (andClose){
        // float it on the left
        andClose.style.float='left';
        // fade text
        andClose.style.color="#aaa";
        // or you can just remove the button altogether:
        // andClose.parentNode.removeChild(andClose);
    }
}
leftClose();

This is what it looks like to me:

[epatters]

Nice one! Thanks.

[klonuo]

I noticed same behaviour with visvis. In this example:

In [1]: import visvis as vv
In [2]: app = vv.use()
In [3]: f = vv.clf()
In [4]: a = vv.cla()

after input [3] IPython console slows down multiple times.

Obviously regardless matplotlibrc backend settings

[klonuo]

Just noticed that visvis default backend is set to Qt4, and when set to other no slowdowns
So this seems like 'expected behavior' then

[epatters]

I confirmed this problem in my Windows 7 VM. I should add that I also experience this on my OS X machine, and though the delay is much less noticeable, I do not think I am imagining it.

Notice this comment from the PyQt4 docs that I linked above: "The installation of an input hook can cause problems for certain applications (particularly those that implement a similar feature using different means)."

I have no idea how to go about making these input hooks play nicely together.

[epatters]

Actually, further testing shows that the delay is just as bad on OS X as Windows. As it stands now, the interaction between pyreadline and PyQt4's input hook makes the terminal IPython effectively useless when using PyQt4 (this includes the Qt pylab backend).

[takluyver]

Is there anything we can do about this before 0.11? Do we need to put a warning in the docs (or in the code)?

[minrk]

That's pretty rough, though at least Windows is the only place that pyreadline is normally used. I don't see any issues when using real readline on OSX.

[epatters]

Hmm. If I run ipython pylab=qt on OS X and then plot something, I experience the delays.

[klonuo]

On Fri, 1 Jul 2011 12:46:33 -0700 takluyver wrote:

Is there anything we can do about this before 0.11? Do we need to put a warning in the docs (or in the code)?

Maybe just fallback on Tk or WX (if user has WX installed), as there
seems no point in running ipython console with Qt4 backend on Windows

[minrk]

Weird, I can't cause any problems at all.

What's your Python, matplotlib, Qt, etc. situation?

[epatters]

My configuration is:

Python - 2.7.1
IPython - master branch
matplotlib - 1.0.1
Qt - 4.7.3
PyQt - 4.8.2

[epatters]

OK, this is even weirder than I thought. Only sometimes do I experience the 2-3 second delays. Sometimes it appears to work fine.

I am thoroughly baffled.

[minrk]

The only major difference I have is I'm using system Python 2.6, but I wouldn't expect that to make a difference. I don 't particularly want to go through the bother of building pyqt for 2.7, but maybe I have to.

I've tried both matplotlib 1.0.1 and git master with no change.
Do you have readline, as installed by 'easy_install readline', or something else?

[fperez]

@epatters, just for my understanding: you're seeing the slowdown happening with normal readline, right? Pyreadline is win32-only, if I remember correctly...

[epatters]

I have a readline module:

In [4]: import readline

In [5]: readline
Out[5]: module 'IPython.utils.rlineimpl' from '/Users/epatters/Local/IPython/IPython/utils/rlineimpl.pyc'

I do not have pyreadline installed.

[minrk]

That will be reported no matter what readline you have. You need to check readline._rl to see the real module if you are in IPython.

@fperez, yes pyreadline is win32 only, but I could swear that I have installed it accidentally on OSX ages ago when I didn't know what name to easy_install.

[SirVer]

I'd like to reopen this issue (but I do not have permission to do so) because I have trouble with gui qt in a very recent clone (65ac74e; yesterday evening). I noticed slugginesh behavior on the ipython shell on two Mac OS X 10.6 Systems and did some experiments.

I used the following unscientific benchmark: I copied and quickly pasted a long text (4500 chars, all one line) in two shells - both ran ipython default profile but in one of them I ran a %gui command before pasting. First experiment was with %gui qt and I noticed that the other ipython shell always finished ~.5 seconds before the gui ipython - even if i pasted in the gui shell first. I then repeated the experiment using %gui osx and the gui shell was as quick as the non gui shell.

The slugginess really affects working especially when walking through the history: I very often press a twice because the reaction comes notably late to the shell.

I noticed that replacing timer.start(50) in lib/inputhookqt4.py:91 with timer.start(1) fixed the problem for me - but upped CPU usage from <1% to > 30%. Also, I really doubt that I would notice a reaction time of 50ms though - I rather think that there is more delay than just those 50ms. Maybe some with more qt experience could help out. I find this code really hard to get - starting and quitting the app periodically doesn't seem like a canonical way of doing things and imho could have implementation detail dependent side-effects on some OSes.

Another fix that works for me like a charm without breaking any functionality (tested using --profile=pylab in this case) is to completely kill the whole block starting in line 88: if not stdin_ready():
In my understanding, my QT windows should no longer react now because the input hook is never called when I do no type inside the ipython shell. But that is not the case, the windows react and there is no felt sluggines in the shell anymore; also the CPU usage remains < 1 %. If I print something in the inputhook, I see that it is called ~10-20 times per second, so it seems good enough to leave the QTApp some 300ms to work its events every time the inputhook is called. I therefore suggest this as fix: https://github.com/SirVer/ipython/commit/8af05fa52f528f782a721191e0c562e4e49e7f10

Below are the versions of one of the boxes (I do not have the other computer around atm):

ipython github 65ac74e
OS X 10.6.8
Python 2.7.1 (build with homebrew)
matplotlib - 1.0.1
PyQt - 4.8.3
Qt - 4.7.1

[fperez]

@SirVer, I've just reopened it. Thanks for all the detailed info, we should be able to make some progress with this...

[cboos]

I just tested 8af05fa, but as I expected this kills the interactivity on Windows as the hook is not called when the process is idle on that platform (well, I must admit I have no idea what happens on a Mac and can't investigate either as I don't have access to one).

[cboos]

... but the interactivity is indeed preserved on Linux! This is due to raw_input, which calls PyOS_Readline / call_readline, and finally readline_until_enter_or_signal which calls PyOS_InputHook in a loop, temporizing every 0.1s (well, this is probably old news for the pyreadline authors around ;-) ).

If I disable the readline extension, then the behavior seen is the same as on Windows (the change kills the interactivity with Qt).

Also, with only that change a CTRL+C is now simply ignored, not sure why at this point.

[SirVer]

I have to add that my 'patch' will indeed reduce the respeonsiveness of (i.e.) matplotlib plot windows - so it is not golden after all but still significantly better than having a lagging shell. I will try out every suggestion on mac os x that is provided - I feel like I cannot provide many suggestions due to lack of knowledge about readline and qt

[fperez]

I don't have really much to offer, other than to thank you guys for pounding on this. It has proven to be a very, very stubborn and hard to fix bug and has most everyone stumped. So even partial progress on it would be very welcome.

[cboos]

So to summarize that part of the problem:

• when readline is used on non-windows platform, the inputhook is called repeatedly by readline (which gives the impression that a simple qApp.processEvents() in the inputhook is enough for guaranteeing interactivity) but readline also wait for 100ms when there's no keypress (which must be what reduces the responsiveness in matplotlib widgets)
• OTOH, if only the inputhook is in control, the current version of the code doesn't immediately exit when a character is pressed so there's a 100ms delay for detecting the first character; the remaining readable characters are taken into account immediately but in case of copy/paste of a long text I can imagine this get chunked and that we see many times the 100ms delay

I think the solution is simply to do what the original PyQt qt_input_hook did in the non-Windows case, add a socket notifier on stdin for quitting the qApp. I'll take on a patch shortly...

[minrk]

@SirVer - can you reproduce the problematic behavior with current (1.1.0) matplotlib, or better yet, without matplotlib at all, just using --gui qt?

[cboos]

Note that I've reproduce the problem as well with just --gui qt, but my idea to add a socket notifier didn't help, I got the same extra delay. I think that what might help is to revert to using the original pyqtRestoreInputHook approach in this situation (non-windows + readline + PyQt). See 7f643b3 for details.

But there's worse: while testing this on Windows with --gui qt, it seems one can "freeze" ipython when pasting a moderately sized text (1561 characters if that rings a bell for someone; no idea myself yet). Disabling pyreadline makes no differences, we have a freeze in either case. Without --gui qt, I can paste a 10k text without trouble (sloooow though with pyreadline, instantaneous without). It seems that this bug predates my changes though, as I have the exact same effect on a3229f5 (the parent cset of my patch series).

Also, note that this "freeze" only concerns the console input, Qt windows remain responsive.

[rkern]

For what it's worth, an EPD customer has seen similar problems on Windows XP with the WXAgg backend to matplotlib. Changing the inputhook constants do not seem to make any noticeable difference. I have not been able to replicate the wx slowness myself on Windows 7 (I can replicate the Qt slowness, though).

[SirVer]

sorry for the late feedback, I was travelling and didn't bother doing my email. Yes, I can also see the problem just with using --gui qt.

[Proletariat99]

Here is a VERY simple newbie solution for someone brand new to python/iPython that may work for some of you newbies like me:

Here's my version info:

• Python 2.7.2 -- EPD_free 7.2-2 (32-bit)
• ipython --version 0.12
• matplotlib 1.1.0
• numpy 1.7.0
• scipy 0.11.0 maybe?

I just created a new profile using the startup option '' and that seemed to do the trick. Utterly newb solution, but it worked for me and may work for you.

For you seasoned devs, here's some info that may help:

Before I found the hidden nanolink at Enthought, I struggled with the install for quite a while. After attempting to install manually a variety of ways, I think I had a very confused python install. I'm guessing that I had something in there that called a problematic input hook that was causing the problem. I wasn't able to find anything, but I honestly didn't look that hard yet.

Here are just a few of the misguided things I did while trying to install:

• gcc manual install (was trying to avoid xcode - failed - no compiler)
• clang install (failed)
  • llvm manual install (clang dependency)
    • freetype2 manual install (llvm dependency)
• Finally gave in to xcode install (hate xcode -- big, closed, clunky)
• Heard "Enthought" enough to realize they HAD to have a free link somewhere that had Numpy/Scipy/Matplotlib as a free package. Went back and scoured their site until I found it.

So then I finally created a new profile and it fixed the super slowness.

Hope that helps someone, somewhere.

Cheers.

[takluyver]

So does that mean that, on a Mac, simply creating a profile solved the
unresponsiveness issue for you? That's a bit odd, because when you
create a profile, it just writes files with all the settings commented
out. But it might give someone something to work with. Thanks, Dave.

[fperez]

I'm sorry to do this, but I'm downgrading this to 'high' priority from 'blocker', simply because we can't realistically block the entire release on an issue we have no clue how to fix.

It's a real bummer, and I hope we'll be able to track what's going on at some point, but absent any progress, we can't stall a release on this. And in fact, we were violating the 'blocker' marker already since we'd bumped this from 0.11 to 0.12 to 0.13, so if anything I'm just being upfront about what we've already done.

[indiajoe]

I would like to report that i have been facing this problem for a long time in my gnu/linux too. Surprised to see it has been reported only in Windows.

---

In short following are the specifications:
Linux Kernel : 3.5.2
Distro: Porteus ( https://en.wikipedia.org/wiki/Porteus_%28operating_system%29 )
Desktop: Trinity (aka. fork of KDE 3.5 )
Python version : 2.6.4
IPython version : 0.12
I run Ipython in terminal by the simple "ipython" command. (no extra arguments)

---

Description of problem:
This usually happens when i leave Ipython terminal open for a few days.
I typically have the numpy, scipy and matplotlib modules loaded.
The machine doesnot gets slow. Infact the commands run by Ipython are also not slow.
Only the keyboard input is slow.
Even if I copy-paste some content into Ipython, it prints each pasted letter one by one with a uniform delay.
While this pasted content is being printed slowly on screen or when i use backspace to remove it, we can see the CPU percentage use of Ipython shooting significantly up in htop.
If I exit the session, and start again, everything works fine as usual.

---

[oliford]

I and my office mate both see this bug in linux too. So here's our experiences, in case it helps isolate it:

I'm using XUbuntu 12.04 (Xfce), ipython 0.12.1 and my office mate also on Linux Mint with ipython 13.1-rc2.
We both see it slowing only the keyboard input (including just moving the cursor) and pasting after use of matplotlib and numpy for some time. I do also use scipy but my officemate doesn't. We're both using the GTKAgg backend.

It doesn't occur immediately, but after some time with use of plotting etc.
We both see significant disk activity during moving the cursor back and forth (as in #1184, perhaps I should have posted this there? I don't want to double post though).

[oliford]

I've found that our problem is exactly that in matplotlib/matplotlib#1530 relating to writing .local/share/recently-used.xbel after opening the GTK (maybe also Qt) save figure dialog once (but not necessarily saving anything). I'll wander over there now.

[danielballan]

I have the same experience as @oliford on Ubuntu 12.04, iPython 0.14.dev, GTKAgg.

[danielballan]

One man's testimonial: I resolved this issue by installing the dev version of matplotlib. Perhaps this commit in this issue cited by @oliford did the trick.

[minrk]

Based on @danielballan's feedback, I would be inclined to close.

[ivanov]

closing based on the conversation around matplotlib/matplotlib#1530.