redirect_in/redirect_out should be constrained to windows only #775

Closed
wants to merge 2 commits into
from

4 participants

@ivanov
IPython member

creating a PIPE for stdin breaks unicode symbols in e.g. !ls magic when inside
qtconsole (at least under Ubuntu)

To verify, make a new file or directory with a unicode character, say mkdir
tmp€

Now in qtconsole, ls will list that directory as tmp���/ in trunk. This PL fixes the
issue.

In Ubuntu 10.04, I see no issues with stdin being invalid when a process is
backgrounded (neither via & at the end of ipython command, nor via
Ctrl-Z followed by bg), and given the nature of redirect_out, and
the Popen bug referenced, I presume this may be a Windows-only issue. So I moved
all of the redirect_in/redirect_out logic to the sys.platform == 'win32'
section.

@epatters - can you take a look at this and verify that it still works on
windows as intended?

update

The original issue described is also solved by #770 without having to remove the stdin PIPE.
Evan says the stdin PIPE solves deadlock issues that arise on OS X, so at least that portion of the change is not windows-only (but is it OS X only?)

@minrk
IPython member

I believe this is actually introduced by recent merge of PR #769 (reopened after merge as #770), which was too aggressive about making bytes safe for JSON (we previously considered bytes safe for JSON, which would usually work on posix, because JSON's default is utf8, as is the default terminal encoding on OSX and Linux, but it wasn't actually safe, causing errors on Windows, or if you print latin1 byes on Linux, etc.).

When trying to be safe about encodings, it's rather challenging when we are starting subprocesses that aren't hooked up to any terminals.

@ivanov
IPython member
@epatters

@ivanov, this is not Windows-specific. As noted in my comments, the Qt console can deadlock when run as a background process. This happens every time for me under OS X (and has been reported by several others). It may be safe for certain Linux configurations, but it is not safe in general.

I can experiment under Linux, but surely there is a better solution for Unicode support than this?

@minrk
IPython member

In these, stdin.encoding is None, and pretty much all of our decoding uses enc=sys.stdin.encoding or sys.getdefaultencoding(), which will return ASCII, so any unicode characters get clobbered.

If we change these cases to sys.stdin.encoding or locale.getpreferredencoding() or sys.getdefaultencoding(), then we should get more sensible, less conservative behavior.

@minrk
IPython member

See PR #770 for a fix using locale in between stdin.encoding and getdefaultencoding. It should fix this issue (and should respect codepages on Windows, etc). It addresses how all bytes objects get to the frontend in general, not just from subprocesses.

@ivanov
IPython member

ok, with #770, I can confirm that ls with unicode filenames works even with PIPE'd stdin, thanks @minrk

I still think that redirect_out should be constrained to the windows section, given that it checks if its running pythonw.exe

shall i reformulate the PL for that? I may get my hands on a Mac OS X platform tomorrow to look into the deadlock issue @epatters describes.

@epatters

@ivanov, I'm glad Min's fix worked for you. Even so, the stdin redirection is a hack, and I would be glad to see it replaced with something else, so feel free to experiment.

@fperez
IPython member

Hey folks, I've just merged #770. I admit I haven't fully grokked the discussion here, what should we do next?

@epatters

I think we can close this, since the PR is not suitable for merging and is really just a workaround for a bug that was apparently fixed (according to Paul's last post).

@fperez
IPython member

@ivanov, does that sound good then? Should we just close this one?

@ivanov
IPython member

sounds good to me, for now, i'll open another one if i ever get around to tracking down the osx issue Evan described

@ivanov ivanov closed this Sep 15, 2011
@fperez
IPython member

Thanks, @ivanov! This helps us keep the PR page reasonably clean so we can focus on the stuff that's really active.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment