Fix for #1688, traceback-unicode issue #1715

jstenar · 2012-05-09T19:20:52Z

PR to work on a fix for #1688. Right now there is one commit with a quick and dirty solution using unicode_literals from __future__ and two casts to unicode. No new test failures were noted.

There are more locations where we will have errors. Any calls to pycolorize needs to make sure to use only unicode. One example is docstring printing using ?

I could add casts in pycolorize instead, should I do that?

takluyver · 2012-05-09T19:38:29Z

IPython/core/ultratb.py

@@ -69,7 +69,7 @@
 #  the file COPYING, distributed as part of this software.
 #*****************************************************************************

-from __future__ import with_statement
+from __future__ import with_statement, unicode_literals


We can get rid of with_statement here - we already dropped Python 2.5 support, and it's native in 2.6+.

Thomas Kluyver skrev 2012-05-09 21:38:

We can get rid of with_statement here - we already dropped Python 2.5 support, and it's native in 2.6+.

ok I'll remove it

takluyver · 2012-05-09T19:51:47Z

We need to be a little bit careful with unicode_literals, because that converts every string in the file, which can have unintended side effects. I've had a scan down the file, and there are no obvious problems, but I'd like us to do some manual testing of this before it lands, because these modules are poorly covered by automated tests.

takluyver · 2012-05-09T19:53:49Z

Test results for commit ca15c23 merged into master
Platform: linux2

python2.7: OK
python3.1: OK (libraries not available: matplotlib pymongo qt wx wx.aui zmq)
python3.2: OK (libraries not available: pymongo wx wx.aui)

Not available for testing: python2.6

takluyver · 2012-05-09T20:01:49Z

For manual testing, three commands I'd like to check work:

%pycat foo.py   # use any reasonably complex Python file you have to hand
1/0                   # trigger a regular traceback
raise = 2           # and a SyntaxError traceback

The focus is on Python 2, since this shouldn't change anything for Python 3. Also, cycle through the three traceback modes using %xmode and repeat the last two tests.

takluyver · 2012-05-09T20:11:16Z

It all seems to be OK on Linux, with Python 2.7.3 and with 2.6.7.

jstenar · 2012-05-10T17:58:03Z

In magic_paste text is sent to colorize from the clipboard. What happens on non-windows platforms? Will clipboard_get return a str or unicode. If str what should I assume about encoding?

takluyver · 2012-05-10T22:21:33Z

I think some of the new functionality in IPython.utils.io is redundant with stuff in IPython.utils.openpy - can you look into merging it into that module?

This PR has also picked up a conflict with master, so we'll need to rebase it before we can merge.

takluyver · 2012-05-10T22:31:01Z

clipboard_get() on Linux appears to retrieve misformed unicode, it looks like this is a bug in Tkinter. On OS X, it's calling a subprocess, so it will likely be in bytes; that should really be fixed within clipboard_get.

takluyver · 2012-05-10T22:57:11Z

I've filed a Python issue for the problems with Tkinter clipboard_get(): http://bugs.python.org/issue14777

jstenar · 2012-05-11T07:23:24Z

I'll look into it. I'm travelling so it will not be before sunday.

11 maj 2012 kl. 00:21 skrev Thomas Kluyver reply@reply.github.com:

I think some of the new functionality in IPython.utils.io is redundant with stuff in IPython.utils.openpy - can you look into merging it into that module?

This PR has also picked up a conflict with master, so we'll need to rebase it before we can merge.

Reply to this email directly or view it on GitHub:
#1715 (comment)

jstenar · 2012-05-14T16:43:33Z

Thomas Kluyver skrev 2012-05-11 00:21:

I think some of the new functionality in IPython.utils.io is redundant with stuff in IPython.utils.openpy - can you look into merging it into that module?

This PR has also picked up a conflict with master, so we'll need to rebase it before we can merge.

I have done a rebase

jstenar · 2012-05-14T17:25:04Z

Thomas Kluyver skrev 2012-05-11 00:31:

clipboard_get() on Linux appears to retrieve misformed unicode, it looks like this is a bug in Tkinter. On OS X, it's calling a subprocess, so it will likely be in bytes; that should really be fixed within clipboard_get.

I have added casts to unicode for clipboard_get on Linux and OSX. I used
DEFAULT_ENCODING.

jstenar · 2012-05-15T19:14:10Z

How do I use the demo mode? In the docs there is a reference to docs/examples/core/demo-exercizer.py but that file seems to be gone.

I think demo mode is broken with this PR as is but I do not know how to use it.

takluyver · 2012-05-15T20:16:35Z

Is demo mode the same thing as this? http://ipython.org/ipython-doc/dev/interactive/reference.html#interactive-demos-with-ipython

takluyver · 2012-05-15T20:22:15Z

IPython/utils/io.py

@@ -14,6 +14,7 @@
 #-----------------------------------------------------------------------------
 # Imports
 #-----------------------------------------------------------------------------
+import re


This is added with no other changes to the file - I guess it's a leftover?

jstenar · 2012-05-15T20:36:31Z

Thomas Kluyver skrev 2012-05-15 22:16:

Is demo mode the same thing as this? http://ipython.org/ipython-doc/dev/interactive/reference.html#interactive-demos-with-ipython

yes that description should be enough. I found this one
http://ipython.org/ipython-doc/rel-0.12.1/api/generated/IPython.lib.demo.html?highlight=demo#IPython.lib.demo

takluyver · 2012-05-17T12:45:05Z

I've got the problem with Tkinter fixed for Python 3.3/3.2.4/2.7.4. I'm now a CPython contributor! :-)

jstenar · 2012-05-31T17:08:09Z

I just rebased from master 2caea25 after the magics PR.

jstenar · 2012-05-31T17:13:31Z

The magics refactoring broke several magic commands (pdef, pdoc, psource, pfile) commit a71d86a fixes this

takluyver · 2012-05-31T19:46:21Z

IPython/core/debugger.py

@@ -361,9 +373,8 @@ def format_stack_entry(self, frame_lineno, lprefix=': ', context = 3):
                      and tpl_line_em \
                      or tpl_line
            ret.append(self.__format_line(linetpl, filename,
-                                          start + 1 + i, line,
+                                          start + 1 + i, line.decode(encoding, errors="replace"),


Are you sure about this? I expect in Python 3, line will already be str, so decoding it again will fail.

Thomas Kluyver skrev 2012-05-31 21:46:

@@ -361,9 +373,8 @@ def format_stack_entry(self, frame_lineno, lprefix=': ', context = 3):
and tpl_line_em
or tpl_line
ret.append(self.__format_line(linetpl, filename,

start + 1 + i, line,

start + 1 + i, line.decode(encoding, errors="replace"),

Are you sure about this? I expect in Python 3, line will already be str, so decoding it again will fail.

fixed in 98fd6f8

takluyver · 2012-05-31T20:02:21Z

Test results for commit a71d86a merged into master
Platform: linux2

python2.7: OK
python3.1: OK (libraries not available: cython matplotlib pymongo qt wx wx.aui zmq)
python3.2: OK (libraries not available: cython pymongo wx wx.aui)

Not available for testing: python2.6

fperez · 2012-06-10T21:46:07Z

Hey guys, just touching bases on this. How far do you think we are from merging it? We have so many PRs in flight that I'm a bit worried and don't want to create multiple conflicts everywhere... We also need to begin thinking about 0.13 sometime soon...

takluyver · 2012-06-10T23:22:29Z

IPython/core/debugger.py

@@ -351,6 +367,10 @@ def format_stack_entry(self, frame_lineno, lprefix=': ', context = 3):

        start = lineno - 1 - context//2
        lines = linecache.getlines(filename)
+        try:
+            encoding, _ = openpy.detect_encoding(_readline(lines))


I suspect Python 3 linecache will take care of decoding lines for us, and this will go wrong.

The test run appears to confirm my suspicion.

fix to handle the case when len(lines)<2

when calling %run -d the filename available for pdb is <string> which is not a proper filename. %run now adds a _exec_lines attribute containing the correct filename.

Use read_py_file to get python source in unicode format.

Fix to interpret str docstrings with the encoding of the sourcefile.

Fix to interpret source of object with the encoding of the sourcefile.

travisbot · 2012-08-27T06:11:42Z

This pull request passes (merged ca43b98 into 011222a).

Most files in linecache are stored with the filename as a bytestring (py2.x). When calling getlines with a unicode filename there will be implicit conversions using ascii that fails. This is avoided by an explicit conversion using the filesystem encoding before calling getlines.

travisbot · 2012-08-27T17:18:57Z

This pull request passes (merged 50c631d into 011222a).

For instance Windows IO error messages are localized and may contain non-ascii chracters.

travisbot · 2012-08-29T06:11:58Z

This pull request passes (merged 8ed4178 into 011222a).

takluyver · 2012-09-03T14:23:57Z

Test results for commit 8ed4178 merged into master (c678a87)
Platform: linux2

python2.7: OK (libraries not available: azure oct2py pymongo wx wx.aui)
python3.2: OK (libraries not available: azure oct2py pymongo wx wx.aui)

Extra args: ['--all']
Not available for testing: python2.6

takluyver · 2012-09-04T12:49:30Z

OK, we reviewed this at Euroscipy and fixed things up. I think it should go in so we can start finding any odd consequences. @jstenar @fperez @ellisonbg @minrk @bfroehle , I'll merge this tomorrow if no-one objects.

Fix for #1688, traceback-unicode issue

Add git-mpr, to merge PR(s) from github just by number(s). Inspired by git-mrb and test_pr, I thougth it would be usefull to be able to merge PR only by giving their number. Hence `git merge-pull-request`or `git-mpr`. usage: ```bash $ git mpr -h ~/ipython/tools usage: git-mpr [-h] [-l | -a | -m [pr-number [pr-number ...]]] Merge (one|many) github pull request by their number. If pull request can't be merge as is, cancel merge, and continue to the next if any. optional arguments: -h, --help show this help message and exit -l, --list list PR, their number and their mergeability -a, --merge-all try to merge as many PR as possible, one by one -m [pr-number [pr-number ...]], --merge [pr-number [pr-number ...]] The pull request numbers ``` examples : ```bash $ git mpr --list * ipython#1758 [√]: test_pr, fallback on http if git protocol fail, and SSL errors... * ipython#1755 [√]: test for pygments before running qt tests * ipython#1715 [√]: Fix for ipython#1688, traceback-unicode issue [...] * ipython#1493 [√]: Selenium web tests proof-of-concept * ipython#1471 [ ]: simplify IPython.parallel connections and enable Controller Resume * ipython#1343 [ ]: Add prototype shutdown button to Notebook dashboard * ipython#1285 [√]: Implementation of grepping the input history using several patterns * ipython#1215 [√]: updated %quickref to show short-hand for %sc and %sx ``` PR number, mergeability and title Quite slow, as it does 1 api call by PR, since api does not give mergeability anymore if you ask for the list of all PRs at once. merge one or more PR (skip the ones with conflict and present a nice list to copy and past to do it by hand) ```bash $ git mpr --merge [pr-number [pr-number ...]]] [...] ************************************************************************************* the following branch have not been merged automatically, considere doing it by hand : PR 1630: git pull https://github.com/minrk/ipython.git mergekernel PR 1343: git pull https://github.com/takluyver/ipython.git notebook-shutdown PR 1715: git pull https://github.com/jstenar/ipython.git ultratb-pycolorize-unicode PR 1732: git pull https://github.com/fperez/ipython.git cellmagics PR 1471: git pull https://github.com/minrk/ipython.git connection PR 1674: git pull https://github.com/mdboom/ipython.git notebook-carriage-return ************************************************************************************* ``` And last, ``` git mpr --merge-all ``` That is pretty self explainatory

Fix for ipython#1688, traceback-unicode issue

takluyver reviewed May 9, 2012
View reviewed changes

takluyver mentioned this pull request May 9, 2012

Fix for fake filenames in verbose traceback #1716

Merged

takluyver reviewed May 15, 2012
View reviewed changes

takluyver reviewed May 31, 2012
View reviewed changes

takluyver reviewed Jun 10, 2012
View reviewed changes

jstenar and others added 16 commits August 27, 2012 08:06

change explicit decode to cast_unicode

9acb302

replacing pop with _readline

9f814d0

fix to handle the case when len(lines)<2

fix for list pdb listcommand, broken for %run -d

82c96fc

when calling %run -d the filename available for pdb is <string> which is not a proper filename. %run now adds a _exec_lines attribute containing the correct filename.

Compare against python2 str version

486d291

Fix to %pfile to ensure unicode handling

c6aa16a

Use read_py_file to get python source in unicode format.

Fix additional unicode problem in ultratb.py

0665343

Fix to show encoding-cookie in %pfile

4ac629e

Unicode fix for docstrings in pinfo magic

4442e9a

Fix to interpret str docstrings with the encoding of the sourcefile.

Unicode fix for docstrings in pinfo2 magic

c570f16

Fix to interpret source of object with the encoding of the sourcefile.

Fix getting unicode lines in IPython.core.debugger.

e5031dd

Fixes for producing unicode tracebacks.

5adf07f

Add test for non-ascii tracebacks.

e79c247

Fix for going into the debugger with non-ascii filenames.

307d4a0

Fix unicode problem with SyntaxError traceback

0ca5179

Switching to use ulinecache on SyntaxError traceback

7e2cf8f

Adding value.text fallback if ulinecache returns ''

ca43b98

jstenar added 2 commits August 27, 2012 19:02

Make magic pycat print encoding_cookie

fcdc9f8

Cast exception.value to unicode.

8ed4178

For instance Windows IO error messages are localized and may contain non-ascii chracters.

jstenar added a commit that referenced this pull request Sep 6, 2012

Merge pull request #1715 from jstenar/ultratb-pycolorize-unicode

5d645a2

Fix for #1688, traceback-unicode issue

jstenar merged commit 5d645a2 into ipython:master Sep 6, 2012

takluyver mentioned this pull request May 30, 2013

Magic %paste not working in Python 3.3.2. TypeError: Type str doesn't support the buffer API #3386

Closed

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request ipython#1715 from jstenar/ultratb-pycolorize-unicode

4257526

Fix for ipython#1688, traceback-unicode issue

takluyver mentioned this pull request Oct 23, 2017

Simplify oinspect.getdoc. #10866

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix for #1688, traceback-unicode issue #1715

Fix for #1688, traceback-unicode issue #1715

jstenar commented May 9, 2012

takluyver May 9, 2012

jstenar May 9, 2012

takluyver commented May 9, 2012

takluyver commented May 9, 2012

takluyver commented May 9, 2012

takluyver commented May 9, 2012

jstenar commented May 10, 2012

takluyver commented May 10, 2012

takluyver commented May 10, 2012

takluyver commented May 10, 2012

jstenar commented May 11, 2012

jstenar commented May 14, 2012

jstenar commented May 14, 2012

jstenar commented May 15, 2012

takluyver commented May 15, 2012

takluyver May 15, 2012

jstenar commented May 15, 2012

takluyver commented May 17, 2012

jstenar commented May 31, 2012

jstenar commented May 31, 2012

takluyver May 31, 2012

jstenar May 31, 2012

takluyver commented May 31, 2012

fperez commented Jun 10, 2012

takluyver Jun 10, 2012

takluyver Aug 5, 2012

travisbot commented Aug 27, 2012

travisbot commented Aug 27, 2012

travisbot commented Aug 29, 2012

takluyver commented Sep 3, 2012

takluyver commented Sep 4, 2012

Fix for #1688, traceback-unicode issue #1715

Fix for #1688, traceback-unicode issue #1715

Conversation

jstenar commented May 9, 2012

takluyver May 9, 2012

Choose a reason for hiding this comment

jstenar May 9, 2012

Choose a reason for hiding this comment

takluyver commented May 9, 2012

takluyver commented May 9, 2012

takluyver commented May 9, 2012

takluyver commented May 9, 2012

jstenar commented May 10, 2012

takluyver commented May 10, 2012

takluyver commented May 10, 2012

takluyver commented May 10, 2012

jstenar commented May 11, 2012

jstenar commented May 14, 2012

jstenar commented May 14, 2012

jstenar commented May 15, 2012

takluyver commented May 15, 2012

takluyver May 15, 2012

Choose a reason for hiding this comment

jstenar commented May 15, 2012

takluyver commented May 17, 2012

jstenar commented May 31, 2012

jstenar commented May 31, 2012

takluyver May 31, 2012

Choose a reason for hiding this comment

jstenar May 31, 2012

Choose a reason for hiding this comment

takluyver commented May 31, 2012

fperez commented Jun 10, 2012

takluyver Jun 10, 2012

Choose a reason for hiding this comment

takluyver Aug 5, 2012

Choose a reason for hiding this comment

travisbot commented Aug 27, 2012

travisbot commented Aug 27, 2012

travisbot commented Aug 29, 2012

takluyver commented Sep 3, 2012

takluyver commented Sep 4, 2012