Fix for incorrect default encoding on Windows.

[dhirschfeld]

Whilst trying out rendering notebooks in a flask app under Apache on Windows I got the below error when simply trying to import SlidesExporter

mod_wsgi (pid=6260): Exception occurred processing WSGI script 'flask_test.wsgi'.
Traceback (most recent call last):
  File "flask_test.py", line 81, in render_notebook
    from IPython.nbconvert.exporters import SlidesExporter
  File "c:\\dev\\code\\ipython\\IPython\\__init__.py", line 47, in <module>
    from .terminal.embed import embed
  File "c:\\dev\\code\\ipython\\IPython\\terminal\\embed.py", line 32, in <module>
    from IPython.terminal.interactiveshell import TerminalInteractiveShell
  File "c:\\dev\\code\\ipython\\IPython\\terminal\\interactiveshell.py", line 25, in <module>
    from IPython.core.interactiveshell import InteractiveShell, InteractiveShellABC
  File "c:\\dev\\code\\ipython\\IPython\\core\\interactiveshell.py", line 59, in <module>
    from IPython.core.prompts import PromptManager
  File "c:\\dev\\code\\ipython\\IPython\\core\\prompts.py", line 138, in <module>
    HOME = py3compat.str_to_unicode(os.environ.get("HOME","//////:::::ZZZZZ,,,~~~"))
  File "c:\\dev\\code\\ipython\\IPython\\utils\\py3compat.py", line 18, in decode
    return s.decode(encoding, "replace")
LookupError: unknown encoding: cp0

A little bit of googling suggests that Windows returns 'cp0' to indicate there is no code page. This fix simply looks for this invalid value and replaces it with something valid. With this change it works for me.

[takluyver]

Can the relevant line have an explanatory comment saying under what circumstances cp0 occurs?

Also, is utf-8 the best default for Windows? Windows doesn't normally use utf-8. In western locales, the default code page is cp1252, I think.

[dhirschfeld]

In my Western European locale cp1252 is the default, but I doubt it is in many other parts of the world so in the absence of any information I thought it might be best to use something that has a chance of working everywhere. It certainly doesn't seem to have any adverse effects for me choosing utf-8 when it should be cp1252. I'll defer to whatever you think is best as I don't have any experience/expertise in this area!

[dhirschfeld]

See also http://stackoverflow.com/questions/17096631/lookuperror-unknown-encoding-cp0 where the user (arbitrarily) chose cp437

[takluyver]

The issue is that the default encoding is supposed to be our best guess for talking to various system things, like running subprocesses. If the system reports cp0, then we don't have much information to go on, but as I understand it, utf-8 is quite unlikely to be the right guess anywhere (on Windows). It's possible to set code page 65001 to use UTF-8, but I don't think that's the default anywhere, and not many users do it manually.

cp1252 certainly won't be correct for everyone, but if we have no other information, I think it's our best guess on Windows.

cp1252, utf-8 and ascii (and most other code pages) are equivalent so long as you stay within ASCII characters (bytes 0 to 127). So if you're an English speaker like me, it will probably work no matter what you pick. To test it, you need to throw in some characters like é or ø or €.

@jstenar : You're the Windows & unicode expert - what's the best default to pick if we don't have any information on the current code page?

[jstenar]

I do not know what can cause cp0 but I wouldn't use utf-8 as a replacement. Perhaps we should generate a warning if cp0 is detected at start and have a way to let the user select a replacement value for cp0. It will be impossible to guess a value that works in all cases.

I would suggest using cp1252 or plain ascii as a default.

[dhirschfeld]

I got my local branch in a little bit of a mess so I rebased on top of master, squashed all the commits into one and forced pushed.

I'm not sure how you'd let the user choose since the error is triggered by simply importing from ipython rather than actually running it. Also, it seems this only crops up when a service account (such as used by a webserver) is used so there may be no human user to ask what the default should be. I did take your advice and add a warning so the user has a chance of detecting the fact that a possibly invalid code page may be being used.

[takluyver]

Yep, I don't think prompting the user is the best path even if it is feasible.

I think this looks good now. We can always make more changes if problems crop up later, but I'm merging it.