unicode exception when using %run with failing script

[jstenar]

When I %run the script below in a path containing non-ascii characters the traceback of a failing script is clobbered by a UnicodeDecodeError in IPython code.

import pdb
def a():
    a=1+2
    1/0
    b=2+4

a()

Correct behaviour when running in ascii path

In [1]: %run err.py
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\utils\py3compat.pyc in execfile(fname, glob, loc)
    166             else:
    167                 filename = fname
--> 168             exec compile(scripttext, filename, 'exec') in glob, loc
    169     else:
    170         def execfile(fname, *where):

C:\python\ipydevel\err.py in <module>()
      5     b=2+4
      6
----> 7 a()

C:\python\ipydevel\err.py in a()
      2 def a():
      3     a=1+2
----> 4     1/0
      5     b=2+4
      6

ZeroDivisionError: integer division or modulo by zero

incorrect behaviour when running in non-ascii path

In [4]: %run err.py
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\interactiveshell.pyc in run_code(self, code_obj)
   2623             self.CustomTB(etype,value,tb)
   2624         except:
-> 2625             self.showtraceback()
   2626         else:
   2627             outflag = 0

C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\interactiveshell.pyc in showtraceback(self, exc_tuple, filename, tb_offset, exception_only)
   1699                     else:
   1700                         stb = self.InteractiveTB.structured_traceback(etype,
-> 1701                                                 value, tb, tb_offset=tb_offset)
   1702
   1703                         self._showtraceback(etype, value, stb)

C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, context)
   1163         self.tb = tb
   1164         return FormattedTB.structured_traceback(
-> 1165             self, etype, value, tb, tb_offset, context)
   1166
   1167 #---------------------------------------------------------------------------

C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\ultratb.pyc in structured_traceback(self, etype, value, tb, tb_offset, context)
   1071             # Verbose modes need a full traceback
   1072             return VerboseTB.structured_traceback(
-> 1073                 self, etype, value, tb, tb_offset, context
   1074             )
   1075         else:

C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\ultratb.pyc in structured_traceback(self, etype, evalue, etb, tb_offset, context)
    917                 frames.append('%s%s' % (level,''.join(
    918                     _format_traceback_lines(lnum,index,lines,Colors,lvals,
--> 919                                             col_scheme))))
    920
    921         # Get (safely) a string form of the exception info

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 26: ordinal not in range(128)

[minrk]

Interesting that this seems Windows-specific, because I get no such issue on a non-ascii path on OSX.

[jstenar]

I believe this is caused by ultratb and pycolorize that are completely unicode ignorant. Perhaps for some reason paths are generated as unicode on windows and str on osx thus not raising the error on osx.

I would like to work on a fix for this by making ultratb and pycolorize unicode aware. Is there any other work currently on this code?

[minrk]

That would be great. No, I don't think there is any other work going on in that code.

[fperez]

I've confirmed this is an issue on linux. It should be very easy to write a test, just make a temp dir with a unicode character in the prefix, put a file with 1/0 inside, cd to it, and call %run on it, catching only the ZeroDivisionError. The test will then fail, and we can get to fixing it.

Does anyone have bandwith to tackle this very soon, so we could hope to ship with this bug fixed in 0.13?

[takluyver]

I'd look at it, but I can't replicate it here - I see the unicode character in the traceback.

[fperez]

Mmh, I was able to replicate it trivially on an ubuntu 12.04 box, with this code:

cd tmp/junk/
mkdir a
mkdir bé
echo "1/0" > a/f.py
echo "1/0" > bé/f.py
cd a/
ipython # run f.py here, all ok

cd ../bé/
ipython  # run here, problem above

see if this helps...

[takluyver]

Bizarre, this is on Ubuntu 12.04 as well:

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/usr/lib/python2.7/dist-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    176             else:
    177                 filename = fname
--> 178             __builtin__.execfile(filename, *where)

/tmp/bé/f.py in <module>()
----> 1 1/0

ZeroDivisionError: integer division or modulo by zero

[fperez]

Mmh... I guess that means I'll have to be the one fixing it :)

[fperez]

OK, I give up, this is very, very weird: I can indeed reproduce a UnicodeError, but a different one than @jstenar is getting, and in my case I only manage to get the problem the first time I run a script. Witness the really weird traceback:

In [1]: pwd
Out[1]: u'/home/fperez/tmp/junk/b\xe9'

In [2]: !cat f.py
1/0

In [3]: run f
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-3-1ce59aa93792> in <module>()
----> 1 get_ipython().magic(u'run f')

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s)
   2150         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2151         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2152         return self.run_line_magic(magic_name, magic_arg_s)
   2153 
   2154     #-------------------------------------------------------------------------

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_line_magic(self, magic_name, line)
   2076                 args.append(sys._getframe(stack_depth).f_locals)
   2077             with self.builtin_trap:
-> 2078                 result = fn(*args)
   2079             return result
   2080 

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/magics/execution.pyc in run(self, parameter_s, runner, file_finder)

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
    188     # but it's overkill for just that one bit of state.
    189     def magic_deco(arg):
--> 190         call = lambda f, *a, **k: f(*a, **k)
    191 
    192         if callable(arg):

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/magics/execution.pyc in run(self, parameter_s, runner, file_finder)
    615                         else:
    616                             # regular execution
--> 617                             runner(filename, prog_ns, prog_ns, exit_ignore=exit_ignore)
    618 
    619                 if 'i' in opts:

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in safe_execfile(self, fname, *where, **kw)
   2463                 if kw['raise_exceptions']:
   2464                     raise
-> 2465                 self.showtraceback()
   2466 
   2467     def safe_execfile_ipy(self, fname):

/home/fperez/usr/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in showtraceback(self, exc_tuple, filename, tb_offset, exception_only)
   1709             # to avoid affecting the exc_info
   1710             try:
-> 1711                 from IPython.parallel.error import RemoteError
   1712             except ImportError:
   1713                 class RemoteError(Exception): pass

/home/fperez/usr/lib/python2.7/site-packages/IPython/parallel/__init__.py in <module>()
     19 import warnings
     20 
---> 21 import zmq
     22 
     23 from IPython.config.configurable import MultipleInstanceError

/usr/lib/python2.7/dist-packages/zmq/__init__.py in <module>()
     36 initthreads.init_threads()
     37 
---> 38 from zmq import core, devices
     39 from zmq.core import *
     40 

/usr/lib/python2.7/dist-packages/zmq/core/__init__.py in <module>()
     24 #-----------------------------------------------------------------------------
     25 
---> 26 from zmq.core import (constants, error, message, context,
     27                       socket, poll, stopwatch, version, device )
     28 

/home/fperez/tmp/junk/bé/socket.pyx in init zmq.core.socket (zmq/core/socket.c:10356)()

/usr/lib/python2.7/dist-packages/zmq/utils/jsonapi.py in <module>()
     48     except ImportError:
     49         try:
---> 50             import simplejson as jsonmod
     51         except ImportError:
     52             try:

/home/fperez/usr/local/lib/python2.7/site-packages/simplejson-1.9.2-py2.6-linux-i686.egg/simplejson/__init__.py in <module>()

/home/fperez/usr/local/lib/python2.7/site-packages/simplejson-1.9.2-py2.6-linux-i686.egg/simplejson/decoder.py in <module>()

/home/fperez/usr/local/lib/python2.7/site-packages/simplejson-1.9.2-py2.6-linux-i686.egg/simplejson/_speedups.py in <module>()

/home/fperez/usr/local/lib/python2.7/site-packages/simplejson-1.9.2-py2.6-linux-i686.egg/simplejson/_speedups.py in __bootstrap__()

/usr/lib/python2.7/dist-packages/pkg_resources.py in <module>()
   2695 
   2696 # Prepare the master working set and make the ``require()`` API available
-> 2697 _declare_state('object', working_set = WorkingSet())
   2698 
   2699 try:

/usr/lib/python2.7/dist-packages/pkg_resources.py in __init__(self, entries)
    433 
    434         for entry in entries:
--> 435             self.add_entry(entry)
    436 
    437 

/usr/lib/python2.7/dist-packages/pkg_resources.py in add_entry(self, entry)
    448         self.entry_keys.setdefault(entry, [])
    449         self.entries.append(entry)
--> 450         for dist in find_distributions(entry, True):
    451             self.add(dist, entry, False)
    452 

/usr/lib/python2.7/dist-packages/pkg_resources.py in find_distributions(path_item, only)
   1688 def find_distributions(path_item, only=False):
   1689     """Yield distributions accessible via `path_item`"""
-> 1690     importer = get_importer(path_item)
   1691     finder = _find_adapter(_distribution_finders, importer)
   1692     return finder(importer, path_item, only)

/usr/lib/python2.7/pkgutil.pyc in get_importer(path_item)
    382         for path_hook in sys.path_hooks:
    383             try:
--> 384                 importer = path_hook(path_item)
    385                 break
    386             except ImportError:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 23: ordinal not in range(128)

But if in the next line I immediately repeat the run f command, then it's fine:

In [4]: run f
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
/home/fperez/usr/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    176             else:
    177                 filename = fname
--> 178             __builtin__.execfile(filename, *where)

/home/fperez/tmp/junk/bé/f.py in <module>()
----> 1 1/0

ZeroDivisionError: integer division or modulo by zero

Somehow this is being triggered by state deep in python's pkg_resources machinery, and the problem only appears once. I have no clue what's going on.

Furthermore, I can only make this happen interactively. All my attempts at writing a test that would also show the problem have failed miserably.

Given that this:

• is much harder to trigger than we thought
• only seems to happen once per session
• appears to be a python problem

I'm going to demote it to having no milestone. There's no point in targeting with a milestone bugs we can't realistically fix, and I've burned already too much time on this.

If anyone can find a reproducible way to cause the problem, great! We'll then have a PR and a fix. But there's no point in keeping it high on the radar otherwise.

[takluyver]

It looks like your traceback is occurring while trying to load simplejson - I don't have that installed, which might explain why I don't see it.

[jstenar]

Fernando Perez skrev 2012-06-26 07:34:

OK, I give up, this is very, very weird: I can indeed reproduce a UnicodeError, but a different one than @jstenar is getting, and in my case I only manage to get the problem the first time I run a script. Witness the really weird traceback:

My first thought on reading that you only see it on the first time you
run the script was that it is a warning that causes a unicode error.
However I do not see anything related to a warning in that traceback.

[minrk]

What happens if you just do: import simplejson?

[minrk]

Can you try #2043?

[fperez]

@minrk, import simplejson is fine. In fact, I tried executing manually just about every frame of that stack trace and could never reproduce the problem standalone. Maddening...

Trying #2043 now.

[fperez]

OK, #2043 works like a charm. I'm going to merge that and assume we're good on this one, but @jstenar, feel free to reopen if you can still cause the problem...

[jstenar]

This is the problem I'm working on in #1715 so I can indeed still reproduce the problem on master (it is fixed in that branch).
So I will reopen the issue.

[jstenar]

Closing after merge of #1715