unicode errors when opening a new notebook

[jstenar]

On windows 7 ipython 731eac3 I get a unicode exception when opening a notebook from a path containing non-ascii characters. I do not get the error if only the notebook and not the path contains non-ascii characters.

Example:

C:\python\ipydevel\notebooks\åäö> ipython notebook
[NotebookApp] Using existing profile dir: u'C:\\Users\\jstenar\\.ipython\\profile_default'
[NotebookApp] Serving notebooks from local directory: C:\python\ipydevel\notebooks\åäö
[NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/
[NotebookApp] Use Control-C to stop this server and shut down all kernels.
[NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js
ERROR:root:Uncaught exception POST /kernels?notebook=b6abbe73-c225-4182-8b86-4c4094480d4d (127.0.0.1)
HTTPRequest(protocol='http', host='127.0.0.1:8888', method='POST', uri='/kernels?notebook=b6abbe73-c225-4182-8b86-4c4094480d4d', version='HTTP/1.1', remote_ip='127.0.0.1', body='', headers={'Origin': 'http://127.0.0.1:8888', 'Content-Length': '0', 'Accept-Language': 'sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4', 'Accept-Encoding': 'gzip,deflate,sdch', 'Host': '127.0.0.1:8888', 'Accept': 'application/json, text/javascript, */*; q=0.01', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22', 'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3', 'Connection': 'keep-alive', 'X-Requested-With': 'XMLHttpRequest', 'Referer': 'http://127.0.0.1:8888/b6abbe73-c225-4182-8b86-4c4094480d4d'})
Traceback (most recent call last):
  File "C:\python27\lib\site-packages\tornado\web.py", line 1021, in _execute
    getattr(self, self.request.method.lower())(*args, **kwargs)
  File "C:\python27\lib\site-packages\tornado\web.py", line 1794, in wrapper
    return method(self, *args, **kwargs)
  File "C:\python27\lib\site-packages\ipython-1.0.dev-py2.7.egg\IPython\frontend\html\notebook\handlers.py", line 352, in post
    kernel_id = km.start_kernel(notebook_id, cwd=nbm.notebook_dir)
  File "C:\python27\lib\site-packages\ipython-1.0.dev-py2.7.egg\IPython\frontend\html\notebook\kernelmanager.py", line 85, in start_kernel
    kernel_id = super(MappingKernelManager, self).start_kernel(**kwargs)
  File "C:\python27\lib\site-packages\ipython-1.0.dev-py2.7.egg\IPython\kernel\multikernelmanager.py", line 98, in start_kernel
    km.start_kernel(**kwargs)
  File "C:\python27\lib\site-packages\ipython-1.0.dev-py2.7.egg\IPython\kernel\kernelmanager.py", line 950, in start_kernel
    **kw)
  File "C:\python27\lib\site-packages\ipython-1.0.dev-py2.7.egg\IPython\kernel\kernelmanager.py", line 919, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "C:\python27\lib\site-packages\ipython-1.0.dev-py2.7.egg\IPython\kernel\launcher.py", line 227, in launch_kernel
    stdin=_stdin, stdout=_stdout, stderr=_stderr, cwd=cwd, env=os.environ)
  File "C:\python27\lib\subprocess.py", line 679, in __init__
    errread, errwrite)
  File "C:\python27\lib\subprocess.py", line 893, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 29-31: ordinal not in range(128)
ERROR:root:500 POST /kernels?notebook=b6abbe73-c225-4182-8b86-4c4094480d4d (127.0.0.1) 13.00ms

[minrk]

test:

In a non-ascii location, try:

import os
from subprocess import Popen
Popen("python.exe", cwd=os.getcwdu())

[jstenar]

>>> import os
>>> from subprocess import Popen
>>> Popen("python.exe", cwd=os.getcwdu())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\python27\lib\subprocess.py", line 679, in __init__
    errread, errwrite)
  File "C:\python27\lib\subprocess.py", line 893, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 19-21: ordinal not in range(128)

[minrk]

smaller tests:

cwdu = os.getcwdu()
# test 1
os.chdir(cwdu)
# test 2
cwdb = cwdu.encode(sys.getfilesystemencoding())
# test 3
os.chdir(cwdb)
# test 4
Popen("time", cwd=cwdb)

[jstenar]

Results of small tests:

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> from subprocess import Popen
>>> cwdu = os.getcwdu()
>>> os.chdir(cwdu)
>>> cwdb = cwdu.encode(sys.getfilesystemencoding())
>>> os.chdir(cwdb)
>>> Popen('echo foo', cwd=cwdb)
<subprocess.Popen object at 0x02960250>

>>> Popen('echo foo', cwd=cwdu)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\python27\lib\subprocess.py", line 679, in __init__
    errread, errwrite)
  File "C:\python27\lib\subprocess.py", line 893, in _execute_child
    startupinfo)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 29-31: ordinal not in range(128)

[minrk]

Okay, so it looks like there is a Windows-specific bug in Python that the cwd kwarg to Popen cannot be unicode, so we should be doing a conditional encode to filesystemencoding() on Python 2 on Windows. One last test: does Popen(cwd=cwdu) work with a proper unicode str in Python 3? With that, I think we should have all of the information we need for a patch.

[jstenar]

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> from subprocess import Popen
>>> import os,sys
>>> cwdu=os.getcwd()
>>> cwdu
'C:\\python\\ipydevel\\åäö'
>>> Popen('echo foo', cwd=cwdu)
<subprocess.Popen object at 0x0309BFB0>

[minrk]

Great, that should be enough info, and #3046 ought to be a sufficient fix.