Writing to sys.stdout fails on Python 3

[takluyver]

Just ran into this with @stefanv. This line:

writer.write(notebook, codecs.getwriter('utf-8')(sys.stdout))

aims to wrap sys.stdout in a unicode interface that will encode using UTF-8 before writing. But in Python 3, sys.stdout is already a unicode interface, so it encodes and tries to write bytes to something that's expecting unicode.

We have Python 2 and 3 compatible code for this in nbconvert - feel free to copy and paste our function:

https://github.com/jupyter/jupyter_nbconvert/blob/3a55808265c8b140cb483c374b5c6ccd464373d0/jupyter_nbconvert/utils/io.py#L12

You'd use this as:

writer.write(notebook, unicode_std_stream('stdout'))

[jni]

Just to clarify, this is a regression between 1.4.0 and 1.4.1, caused by this commit. Would be worth writing a test specifically to catch this when it's fixed.

On 1.4.0:

notedown --match fenced --run markdown/preface.markdown > build_ipynb/preface.ipynb

(no error)

On 1.4.1:

notedown --match fenced --run markdown/preface.markdown > build_ipynb/preface.ipynb
Traceback (most recent call last):
  File "/Users/nuneziglesiasj/anaconda/envs/elegant/bin/notedown", line 9, in <module>
    load_entry_point('notedown==1.4.1', 'console_scripts', 'notedown')()
  File "/Users/nuneziglesiasj/anaconda/envs/elegant/lib/python3.4/site-packages/notedown/main.py", line 265, in cli
    writer.write(notebook, codecs.getwriter('utf-8')(sys.stdout))
  File "/Users/nuneziglesiasj/anaconda/envs/elegant/lib/python3.4/site-packages/ipython-3.1.0-py3.4.egg/IPython/nbformat/__init__.py", line 161, in write
    fp.write(s)
  File "/Users/nuneziglesiasj/anaconda/envs/elegant/lib/python3.4/codecs.py", line 374, in write
    self.stream.write(data)
TypeError: must be str, not bytes

[aaren]

Thanks both.

@takluyver I've just imported from IPython since that's a strong dependency anyway.

@jni yes more tests would be great. I need to be a bit more vigilant about python 3. I don't use it everyday so simple things like this can slip through.

[aaren]

I've released 1.4.2 with this fix.

[jni]

@aaren, thanks for the really quick fix!

I don't use it everyday

Maybe you should! I made the switch just a couple of months ago and I've been pretty happy. =)

[aaren]

@jni I'll switch once thesis is finished...

There is lots of good stuff in python 3 so I want to switch, but at the same time I can't afford to go into a conversion rabbit hole right now!

[takluyver]

Be aware that that function is moving in the reorganisation for IPython/Jupyter 4.0. We've left it exposed at the old import location for backwards compatibility, so your fix should be fine through 4.x releases, but at some point we'll remove it from IPython.utils.io. We're still discussing naming conventions for the new packages, but at present that's located in jupyter_nbconvert.utils.io.

[aaren]

Thanks. When Jupyter is consolidated I'll switch to that.