creating a PIPE for stdin breaks unicode symbols in e.g. !ls magic when inside
qtconsole (at least under Ubuntu)
To verify, make a new file or directory with a unicode character, say mkdir tmp€
Now in qtconsole, ls will list that directory as tmp���/ in trunk. This PL fixes the
In Ubuntu 10.04, I see no issues with stdin being invalid when a process is
backgrounded (neither via & at the end of ipython command, nor via
Ctrl-Z followed by bg), and given the nature of redirect_out, and
the Popen bug referenced, I presume this may be a Windows-only issue. So I moved
all of the redirect_in/redirect_out logic to the sys.platform == 'win32'
sys.platform == 'win32'
@epatters - can you take a look at this and verify that it still works on
windows as intended?
The original issue described is also solved by #770 without having to remove the stdin PIPE.
Evan says the stdin PIPE solves deadlock issues that arise on OS X, so at least that portion of the change is not windows-only (but is it OS X only?)
PIPE for stdin only needed on windows
Moreover, it breaks unicode in e.g. !ls magic
removed unused import (atexit)
I believe this is actually introduced by recent merge of PR #769 (reopened after merge as #770), which was too aggressive about making bytes safe for JSON (we previously considered bytes safe for JSON, which would usually work on posix, because JSON's default is utf8, as is the default terminal encoding on OSX and Linux, but it wasn't actually safe, causing errors on Windows, or if you print latin1 byes on Linux, etc.).
When trying to be safe about encodings, it's rather challenging when we are starting subprocesses that aren't hooked up to any terminals.
The behavior I describe has been around for a while so I think it's independent of #769.
doing ls in a directory with a unicode file worked as of 56d2d8d on Aug 18th, and broke
in adba85c on the next day(which is why I requested that @epatters take a look).
@ivanov, this is not Windows-specific. As noted in my comments, the Qt console can deadlock when run as a background process. This happens every time for me under OS X (and has been reported by several others). It may be safe for certain Linux configurations, but it is not safe in general.
I can experiment under Linux, but surely there is a better solution for Unicode support than this?
In these, stdin.encoding is None, and pretty much all of our decoding uses enc=sys.stdin.encoding or sys.getdefaultencoding(), which will return ASCII, so any unicode characters get clobbered.
enc=sys.stdin.encoding or sys.getdefaultencoding()
If we change these cases to sys.stdin.encoding or locale.getpreferredencoding() or sys.getdefaultencoding(), then we should get more sensible, less conservative behavior.
sys.stdin.encoding or locale.getpreferredencoding() or sys.getdefaultencoding()
See PR #770 for a fix using locale in between stdin.encoding and getdefaultencoding. It should fix this issue (and should respect codepages on Windows, etc). It addresses how all bytes objects get to the frontend in general, not just from subprocesses.
ok, with #770, I can confirm that ls with unicode filenames works even with PIPE'd stdin, thanks @minrk
I still think that redirect_out should be constrained to the windows section, given that it checks if its running pythonw.exe
shall i reformulate the PL for that? I may get my hands on a Mac OS X platform tomorrow to look into the deadlock issue @epatters describes.
@ivanov, I'm glad Min's fix worked for you. Even so, the stdin redirection is a hack, and I would be glad to see it replaced with something else, so feel free to experiment.
Hey folks, I've just merged #770. I admit I haven't fully grokked the discussion here, what should we do next?
I think we can close this, since the PR is not suitable for merging and is really just a workaround for a bug that was apparently fixed (according to Paul's last post).
@ivanov, does that sound good then? Should we just close this one?
sounds good to me, for now, i'll open another one if i ever get around to tracking down the osx issue Evan described
Thanks, @ivanov! This helps us keep the PR page reasonably clean so we can focus on the stuff that's really active.