raw_input redirection to foreign kernels is extremely brittle

[fperez]

In the notebook, open %qtconsole to get a console pointed to the notebook's kernel (which is 'foreign' to the console), then run any script that causes a traceback a few levels deep, and try to run %debug in the console. In more cases than not, the result is a wedged console where the keyboard stops accepting input.

We had major improvements on the raw_input behavior after some fixes made by @epatters to the console, and now it's pretty solid when interacting with its own kernel. But for some reason I don't really understand right now, when talking to kernels it didn't start the behavior is much poorer.

I'm marking it as critical so that at least we have a look at it before 0.12, but if we can't solve it we'll move it to the 0.13 milestone, as it may be harder to fix than we can manage this close to a release, and I don't want to hold 0.12 for any longer than strictly necessary.

[minrk]

Can you provide a sample test case? I can't seem to get this to happen. I don't really see how how it could be different if it's 'foreign', because the code doesn't differ at all except for signaling/shutdown. It would make more sense to me if you see the same bad behavior in either console when two are connected to the same kernel, regardless of which owns it.

We have a few issues outstanding, which taken together are essentially '%debug behavior in the two-process model is pretty bad in general'. The issue being that we don't treat debug specially at all in the frontends - it is just perfectly opaque calls to raw_input, so lots of things don't behave as nicely as in the one-process terminal (history, completion, etc.)

[fperez]

I've seen it fairly frequently when doing %debug and walking up/down the stack a few times. It's the behavior we used to see, where the prompt is printed before some output clobbers it.

I've put error.py here: https://gist.github.com/1367916

and if I %run it in the nb and then from a qtconsole do %debug, after a few instances of u and d I get this:

In [3]: debug
> /home/fperez/scratch/error.py(45)RampNum()
     44     step = (end-start)/(size-1-tmp)
---> 45     result[:] = arange(size)*step + start
     46 


ipdb> u
> /home/fperez/scratch/error.py(62)main()
     61     for i in xrange(reps):
---> 62         RampNum(array_num, size, 0.0, 1.0)
     63     RNtime = time.clock()-t0


ipdb> d
> /home/fperez/scratch/error.py(45)RampNum()
     44     step = (end-start)/(size-1-tmp)
---> 45     result[:] = arange(size)*step + start
     46 


ipdb> l
     40         result[i] = start + step*i
     41 
     42 def RampNum(result, size, start, end):
     43     tmp = zeros(size+1)
     44     step = (end-start)/(size-1-tmp)
---> 45     result[:] = arange(size)*step + start
     46 
     47 def main():
     48     #print 'hi'

     49     size = 6
     50     reps = 5


ipdb> u

ipdb> > /home/fperez/scratch/error.py(62)main()
     61     for i in xrange(reps):
---> 62         RampNum(array_num, size, 0.0, 1.0)
     63     RNtime = time.clock()-t0

At the last one, it's wedged: the output came after the prompt, and typing stops working.

I think we're going to need some logic where, when raw_input is active, we flag that state and track the last written prompt and all characters typed so far, so that if plain output comes out, then we can rewrite everything and reposition the cursor. I don't know if you're familiar enough by now with the cursor handling logic to give this a shot or not... @epatters, do you have any useful pointers/ideas on this one?

[takluyver]

Sometimes when it gets jammed, I see error messages at the terminal where I started it, along the lines of "position 11232 is beyond usable range". I'll see if I can reproduce it.

[takluyver]

Actually, when it gets jammed, with master, I can press Ctrl-C, and get back to a working IPython shell. That's an improvement on 0.11, where once it's jammed, your shell session is done for.

I don't think it's specifically connected to foreign kernels, though - I can get it jammed with a Qt console which started its own kernel normally.

[fperez]

On Tue, Nov 15, 2011 at 11:15 AM, Thomas
reply@reply.github.com
wrote:

Actually, when it gets jammed, with master, I can press Ctrl-C, and get back to a working IPython shell. That's an improvement on 0.11, where once it's jammed, your shell session is done for.

True, though that kills the debug session, for example. Certainly
better than being completely stuck, but highly sub-optimal
nonetheless. And many users may not think of C-C in those
circumstances.

[takluyver]

Really? I'm quite likely to try Ctrl-C if something's got stuck. I agree, though - it's still far from ideal.

[fperez]

Well, oddly enough, in the Qt console, I hadn't thought of using Ctrl-C. If one person has a problem, likely others will too :)

[minrk]

In general, %debug behavior is quite bad in the qtconsole, and I think we
need to look into making it a special case, rather than using generic
raw_input.

Obviously, we should make it harder to get raw_input stuck (I still can't
make it happen on purpose, following your example character-by-character
but I have seen it plenty before), but %debug is a much larger issue, and
probably by far the most complex / common use of raw_input we have.

[fperez]

I'm not sure that %debug is all that special, though. Basically any program that goes into its own REPL built with raw_input will behave similarly. A single raw_input call will typically be OK, but any program that continues to take user input and generates output on it will eventually wedge the console, since the underlying issue is a race condition between the events on the two sockets.

[minrk]

Right, but there is a reason that we don't just forward IPython itself as a raw_input REPL. I agree that it certainly shouldn't break like it does, but I do not think that it is reasonable to expect a repl forwarded to the qtconsole to work as well as it does in the terminal.

[fperez]

Actually, if it's a pure python repl, we might be able to make it work. I'm not completely convinced that we can't make it be robust, but perhaps it really can't be done reliably and we need instead to special-case it...

[minrk]

Well, the reason we should special case debug is history, tab completion, etc. which are not generic to raw_input requests.

A simple repl should probably work, at least if it is sufficiently rudimentary, and if we guarantee that output never gets dumped into the input area, that should go a very long way. I think that is what causes almost all of these problems (raw_input or otherwise).

[takluyver]

Is it possible to maintain a 'stdout point' just before the input prompt,
where anything printed would get inserted?

[minrk]

That's exactly what the 'before_prompt' bit is supposed to do, but it currently doesn't handle the raw_input (internally identified as self._reading) case, only regular prompts.

[fperez]

@minrk, do you think it's easy to adapt the code to handle raw_input as well? It would be awesome if we could make raw_input handling more robust...

[minrk]

It should be pretty close to just changing this line to if before_prompt and (self._reading or not self._executing):, but that change doesn't get it quite right.

I made a couple of tweaks that definitely improve things, but aren't there yet. I don't know the Qt workings well enough to really know where to go from there.

The broader issue is that it should really be impossible to accidentally write output to the text entry area. I don't know enough about the Qt text area to know how we can delineate these things better, but rewriting input is so rare, that it should be the only code that can even try to write to the actively edited area. The _append_foo methods should not even be able to do this.

[fperez]

Agreed. I think @epatters mentioned that since he was using a fully editable widget some of this logic was trickier to implement than if it was a collection of stacked input/output areas (but that would have other issues). But even partial improvements on this front will be super welcome, thanks!

[fperez]

Forn anyone else who's interested, @minrk's changes in #1053 are already a major improvement. Ideally we'd have a completely robust fix, but at list we're on track to something reasonable.