Windows GUI only (pythonw) bug for IPython on Python 3.x

[gbrandonp]

Trying to launch the ipython-qtconsole.exe or ipython-qtconsole-script.pyw via pythonw on Python 3.2 immediately exits, and the Qt console never starts. Running with ipython-qtconsole-script.pyw with plain python.exe works fine (and consumes the console). Fortunately, redirection stdout and stderr allows to get the traceback (well, stderr is where it went), which yields the following:

Traceback (most recent call last):
  File "\Python32\Scripts\ipython3-qtconsole-script.pyw", line 9, in 
    load_entry_point('ipython==0.12', 'gui_scripts', 'ipython3-qtconsole')()
  File "C:\Python32\lib\site-packages\distribute-0.6.24-py3.2.egg\pkg_resources.py", line 337, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "C:\Python32\lib\site-packages\distribute-0.6.24-py3.2.egg\pkg_resources.py", line 2280, in load_entry_point
    return ep.load()
  File "C:\Python32\lib\site-packages\distribute-0.6.24-py3.2.egg\pkg_resources.py", line 1990, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "C:\Python32\lib\site-packages\IPython\__init__.py", line 43, in 
    from .config.loader import Config
  File "C:\Python32\lib\site-packages\IPython\config\loader.py", line 27, in 
    from IPython.utils.path import filefind, get_ipython_dir
  File "C:\Python32\lib\site-packages\IPython\utils\path.py", line 24, in 
    from IPython.utils.process import system
  File "C:\Python32\lib\site-packages\IPython\utils\process.py", line 25, in 
    from ._process_win32 import _find_cmd, system, getoutput, AvoidUNCPath, arg_split
  File "C:\Python32\lib\site-packages\IPython\utils\_process_win32.py", line 30, in 
    from . import text
  File "C:\Python32\lib\site-packages\IPython\utils\text.py", line 30, in 
    from IPython.utils.io import nlprint
  File "C:\Python32\lib\site-packages\IPython\utils\io.py", line 90, in 
    stdin = IOStream(sys.stdin)
  File "C:\Python32\lib\site-packages\IPython\utils\io.py", line 32, in __init__
    raise ValueError("fallback required, but not specified")
ValueError: fallback required, but not specified

I was confused why nobody had seen this before, but come to find out it's a Python 3 vs 2 issue. sys.std* are now None (and will continue to be) for pythonw processes in Python 3.x, while they were file objects with fileno == -2 on Python 2.x. Regular old prints will work when stdout/stderr are None, but they just get discarded immediately, but of course any attribute lookups are going to fail.

Interestingly, running with:
\Python32\pythonw.exe \Python32\Scripts\ipython3-qtconsole-script.pyw > stdout.log 2> stderr.log < empty_file
yielded the qtconsole window, but the prompt never appeared, I assume due redirecting an empty file into stdin (I've no idea what /dev/null is on Windows). I guess in the course of normal script runs, that stdin is a file object that is already closed, or in some other state, so that allows this to work, but doesn't cause it to read input from stdin.

This simple script run with pythonw (on v2 and v3) highlights the difference:

# this will show what the std* file handles are on Python 2.x vs Python 3.x
# noted @ http://bugs.python.org/issue1415
import sys

fname = 'python%d_std_fds.txt' % (sys.version_info[0])
f = open(fname, 'w')

if sys.stdin is None:
    f.write("stdin: None\n")
else:
    f.write("stdin: %i\n" % sys.stdin.fileno())
if sys.stdout is None:
    f.write("stdout: None\n")
else:
    f.write("stdout: %i\n" % sys.stdout.fileno())
if sys.stderr is None:
    f.write("stderr: None\n")
else:
    f.write("stderr: %i\n" % sys.stderr.fileno())
f.close()

[gbrandonp]

By hacking up IPython/utils/io.py variables to std{in,out,err} to make them None if corresponding sys.std{in,out,err} are None, and excepting cases elsewhere that blindly use sys.stding.encoding, and maybe one or two other minor hacks, I get the window up when starting ipython3-qtconsole.exe or pythonw ipython3-qtconsole-script.pyw. However, though I do see the Python version msg and IPython splash message, I never get the prompt.

Tried various other things with the ipython3-qtconsole-script.pyw:

1. ran with pythonw.exe but redirected stdin or stdout or stderr from/to NUL (learned this is windows equiv of /dev/null). immediate failure
2. ran with python.exe and redirectied stdin or stdout or stderr from/to NUL. no change, still get the prompt as normal and everything works fine
3. started ipython3.exe kernel session, then in another cmd window, launched the qtconsole script via pythonw.exe with the --existing option.

• qtconsole works fine, got that cmd window back
• Ctrl+T (new tab w/ new kernel) yields old behavior of splash msg and no prompt

[takluyver]

The banner but no prompt means that the frontend is starting, but it's failing to start or connect to the kernel.

[gbrandonp]

I was able to get better debugging information by installing Mark Hammond's pywin32 modules, and using logging.handlers.NTEventLogHandler to actually get some log data surrounding the failure (I didn't think the logging.handlers.FileHandler would work since both the GUI process and the Kernel process would be using the same name without some plumbing to provide different ones):

--- a/IPython/config/application.py
+++ b/IPython/config/application.py
@@ -19,7 +19,7 @@ Authors:
 # Imports
 #-----------------------------------------------------------------------------

-import logging
+import logging.handlers
 import os
 import re
 import sys
@@ -188,7 +188,7 @@ class Application(SingletonConfigurable):
         if sys.executable.endswith('pythonw.exe'):
             # this should really go to a file, but file-logging is only
             # hooked up in parallel applications
-            self._log_handler = logging.StreamHandler(open(os.devnull, 'w'))
+            self._log_handler = logging.handlers.NTEventLogHandler('ipython')
         else:
             self._log_handler = logging.StreamHandler()
         self._log_formatter = logging.Formatter("[%(name)s] %(message)s")

And what I found was actually a generic Python 3 issue, that is only exposed because the no_stdout and no_stderr are both set to true for the kernel when run pythonw.exe. The problem is the use of file(<filename>) to open files in Python 3. This specific case is in zmq.kernelapp.init_blackhole, but a grep of the source found others. You can repeat by running IPython with Python 3 and supplying either the --no-stderr or --no-stdout options.

Since open() works in both Python 2 and 3, my suggestion (or I can supply a patch, fork and make a pull request) is to replace all file() instances with open(). That still leaves some changes specific to having std{in,out,err} == None for pythonw.exe, but so far, things seem to work fine.

[gbrandonp]

I've got two commits in a fork at https://github.com/gbrandonp/ipython/tree/pythonw-py3k that have things working for me:

1. change file to open
2. deal with the fact that std{in,out,err} are None on py3k with pythonw

[takluyver]

Great, thanks - please make a pull request from that branch so others review it.

[gbrandonp]

Pull request 1245 created

[takluyver]

This should now be fixed by PR #1245 - if anyone still encounters it, let us know.