Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP


Python 3 compatilibility work #663

merged 44 commits into from

4 participants

Thomas Kluyver Fernando Perez Min RK Grahame Bowland
Thomas Kluyver

The eventual aim is to make one codebase that can be installed on Python 3 using 2to3. This doesn't complete that, but it does a substantial chunk of the work, and I think it's a good time to stop and go over it.

Most of the compatibility layer is in a new IPython.utils.py3compat module.

Stuff I know isn't yet done:

  • doctests (might not be needed - 2to3 installation with distribute should translate them automatically, but will it catch IPython doctests?)
  • Script names (e.g. ipython --> ipython3)
  • What do we do about external libraries? In particular, I've made changes to pexpect and simplegeneric for Python 3. I'm happy to push changes upstream, but that could take a while, and for pexpect would probably involve dropping some backwards comatibility (the pexpect website still mentions Python 1.5!)

Full view of the manual changes I've made for Python 3:

Also, I found we had some more-or-less duplicate code for breaking down user input, in inputsplitter and prefilter. I've made them use a common version.

One test is failing at present: IPython.core.tests.test_inputsplitter.test_split_user_input, which attempts to break down the line u"Pérez Fernando". My gut feeling is that it's OK to change the test itself, because there's nothing intrinsically 'correct' about the result it gave before; but perhaps I'm missing something.

Thomas Kluyver

One other problem - the coloured display of the source you get with func?? doesn't appear properly (you just see the first character). It does work with %psource func, though. I'll look into this.

Thomas Kluyver

OK, it now catches errors loading the default config file, and resets exec_lines to an empty list. But when I do ipython3 profile create (or ipython3 profile create python3), it creates the full default config file in .ipython/profile_python3. Doing ipython profile create python3 copies the file across as I'd expect, so I guess it's because python3 is the default profile for ipython3.

Fernando Perez

Hey Thomas, sorry we let this one go stale, but right now it doesn't merge anymore. Could you rebase it against master and push again? I'll be happy to review it then so we can flush it quickly.

Thomas Kluyver

Done. There were just a couple of small collisions. This can now be installed and run on Python 3, although I haven't tested it extensively. With Python 3, it now installs setuptools-style entry points as ipython3 etc. I've patched simplegeneric and sent the diff upstream, but not heard anything yet.

Before merging, we should consider the failing test for splitting user input containing unicode (I think it's alright to change the test, but I'm not sure why it was written).

Other questions I haven't yet addressed, but needn't block merging this: I need to work out a way to convert doctests, which currently cause a lot of failures. Pexpect will need some patches to work in Python 3. We should also give a bit of thought to how the python3 profile should fit in.

Fernando Perez

no, that failing test is indeed a problem. I'm not sure what's going on, but it's really a breakage. In master, we get, correctly:

In [5]: from IPython.core import inputsplitter as isp

In [6]: isp.split_user_input(u'Pérez Fernando')
Out[6]: (u'', '', u'', u'P\xe9rez Fernando')

but with these changes, we get:

In [1]: from IPython.core import inputsplitter as isp

In [2]: isp.split_user_input(u'Pérez Fernando')
Out[2]: (u'', '', u'P', u'\xe9rez Fernando')

So the input is definitely being incorrectly split, because the third output value should be an empty string and we're getting a 'P' in there.

I'm too tired to dig into this one, but we do need to fix this one before merging, not changing the test. If you make any progress let me know...

Thomas Kluyver

I know it changes what happens in that case, but I'm not sure why the behaviour in master is more correct. Non-ascii characters outside a string are invalid in Python 2 anyway, so does it matter how we split up the invalid input?

Thomas Kluyver

More generally, split_user_input looked like it was designed to break out the first valid Python identifier (from a comment: "fpart has to be a valid python identifier"), optionally with IPython escape characters. But it didn't do this very well, so x =1 would be split, but x=1 wouldn't. I tweaked the regex a bit so that it behaved more consistently. So when presented with u'Pérez Fernando', it breaks out P as the first valid Python identifier. Is there some better definition of how it should behave?

Fernando Perez

You're actually right, now that I looked at it more carefully. Go ahead and update the PR so that the test passes.

I've looked it over and I don't see any problems, but since it's pretty big, let me run using this as my default branch for a day and see if I notice any problems. If all looks OK we'll go ahead and merge it.

@mirnk, @ellisonbg: if you have any objection to this one, let us know. It's a fairly large set of changes, but it will make py3 support much easier in the future, and I think Thomas did a great job of isolating the compatibility hacks to very well-defined locations. So unless I find any problems in actual use (I've reviewed it and ran the test suite), I'll merge it in a day or so.

Thomas Kluyver

Thanks, Fernando. I've updated it, and the there are no more test failures on Python 2.

(Tagging @minrk because a typo missed him above. Let us know of any objections to these changes.)

Fernando Perez

Thanks for catching the typo. If @ellisonbg and @minrk have no objections, I'll merge this one tonight (unless I find a glitch in usage today). I've been using it as my normal ipython and haven't seen any problems so far...

Min RK

I just did a PR against @takluyver's branch to localise the Session as bytes/unicode, so we don't have to call encode('ascii') all over the place.

Fernando Perez

Awesome, thanks! And @takluyver: given that this branch does have a few conflicts with the notebook stuff, which we really need for Euroscipy (Stefan and I are planning on using it), let's delay this merge until the nb branch lands. There will be a few small conflicts to fix (just stuff in setupbase, I think) and then we can merge it afterwards.

Thomas Kluyver

OK, great. I'll get Min's bsession changes merged into this branch in the meantime, and possibly investigate converting the doctests.

Fernando Perez

Fantastic. The nb branch is a lot of work and will still take us a day or two to mege, so we may just get to actually merging this when we're in Paris, which isn't the end of the world. Between now and Tuesday I've basically run out of available time for just about anything.

Thomas Kluyver

Rebased after notebook branch merged.

Thomas Kluyver

Rebased again.

takluyver and others added some commits
Thomas Kluyver takluyver Add module for Python 3 compatibility layer. eb1cd97
Thomas Kluyver takluyver Start using py3compat module. ee492f8
Thomas Kluyver takluyver More Python 3 compatibility fixes. 54a1718
Thomas Kluyver takluyver Spelling correction. 7cb775e
Thomas Kluyver takluyver Repair various failures in the test suite. e36691a
Thomas Kluyver takluyver More Python 3 compatibility fixes. 18b2497
Thomas Kluyver takluyver Incidental correction to config file. f707b28
Thomas Kluyver takluyver Add Python 3 profile. c165f2e
Thomas Kluyver takluyver Select default profile according to Python version. 4408c4c
Thomas Kluyver takluyver Fix in preparation for Python 3.3 compatibility. c7f7b48
Thomas Kluyver takluyver Python 3 compatibility for identifiers. 8a38592
Thomas Kluyver takluyver Fix for looking up print function in Python 3. a7e8c70
Thomas Kluyver takluyver Further Python 3 fixes in core. cf41e14
Thomas Kluyver takluyver Add simple implementation of Python 3 style open() d60f657
Thomas Kluyver takluyver Improvements in the code that breaks up user input. 17afbbe
Thomas Kluyver takluyver More Python 3 compatibility fixes in core. ef3358b
Thomas Kluyver takluyver Reuse common code for inputsplitter and prefilter. 76b8481
Thomas Kluyver takluyver Small correction to recent change in oinspect. 74ba4eb
Thomas Kluyver takluyver Convert source to native string before doing ANSI formatting.
Closes gh-665
Thomas Kluyver takluyver Internal machinery should not be printing out error messages. 64eab80
Thomas Kluyver takluyver Make installation with Python 3 possible. b9f6e70
Thomas Kluyver takluyver Backwards compatibility hack to use nested() in Python 3.2 328adae
Thomas Kluyver takluyver Python 3 compatibility in IPython.lib.pretty 5326df5
Thomas Kluyver takluyver Small changes towards supporting Python 3. e57ac64
Thomas Kluyver takluyver Make PickleShareDB inherit from collections.MutableMapping abc 1ec897c
Thomas Kluyver takluyver Fix for nested() c14b594
Thomas Kluyver takluyver Update bundled simplegeneric to version 0.7. bcd1047
Thomas Kluyver takluyver Patch bundled simplegeneric for Python 3 compatibility. d82fb26
Thomas Kluyver takluyver Fixes for UltraTB and PyColorize with Python 3 f866f01
Thomas Kluyver takluyver Fix various problems highlighted by the test suite. 6feb755
Thomas Kluyver takluyver test_oinspect works with Python 3. b3d8f63
Thomas Kluyver takluyver Don't monkeypatch nose in Python 3, as it must be a newer version. fd074fb
Thomas Kluyver takluyver Fix config loader test for Python 3. 1dcb5a6
Thomas Kluyver takluyver Fixes to 5299658
Thomas Kluyver takluyver Various Python 3 fixes in IPython.utils 2467d2e
Thomas Kluyver takluyver Allow 'python install' to work correctly for either Python 2…
… or 3.
Thomas Kluyver takluyver Fix for 'foo?' object inspection in Python 3. 326f739
Thomas Kluyver takluyver Safer loading of default config inside Python 3 config. 6c940b7
Thomas Kluyver takluyver Fix bugs in PyColorize 974068c
Thomas Kluyver takluyver Update split_user_input unicode test. 8c63d48
Min RK minrk add Session.bsession trait for session id as bytes c340839
Thomas Kluyver takluyver Add Trove classifiers for PyPI. a88bf20
Thomas Kluyver takluyver Override input(), not raw_input(), for Qt console in Python 3. 786b191
Grahame Bowland grahame Make PNG images in the Qt console work in Python 3. 307b083
Thomas Kluyver

Rebased again.

Min RK

What still needs to be done here?

If all tests pass on 2.6, it would seem that this is ready for merge, unless there are specific tasks that need doing.

Thomas Kluyver

I'd like to get it merged in. There's some more things to do (doctests, pexpect), but the changes that are already here keep ending up in conflict with changes in master. I should test on 2.6, though - so far I've only been testing on 2.7.

Fernando Perez

OK, I've just run the tests on 2.6, and all looks good. Since we've had this in the wings for so long, I'm merging it now.

Thomas, thanks a lot for the great work! I hope having this now in trunk will make your life significantly easier re. py3 support. Please do yell at us if you see us doing things that make py3 support harder in master: since we don't use it regularly, we tend to slip up easily :)

Fernando Perez fperez merged commit 37dd091 into from
Thomas Kluyver

Thanks, Fernando, that will definitely make my life easier. I'll try to keep checking it.

When you've got a moment, can you also modify the description line at to say it's superseded by work in the main repo. Leave it around for the moment, though, because the diff might still be useful.

Fernando Perez
Min RK

There's now a big problem. We can no longer run any files that are not utf8 encoded. Python itself respects the encoding header, but since we now hardcode the encoding in, all other codings are simply ignored. Fixed in PR #778. I'll merge right away if I get an okay, as it's a very serious issue.

Fernando Perez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 7, 2011
  1. Thomas Kluyver
  2. Thomas Kluyver
  3. Thomas Kluyver
  4. Thomas Kluyver

    Spelling correction.

    takluyver authored
  5. Thomas Kluyver
  6. Thomas Kluyver
  7. Thomas Kluyver
  8. Thomas Kluyver

    Add Python 3 profile.

    takluyver authored
  9. Thomas Kluyver
  10. Thomas Kluyver
  11. Thomas Kluyver
  12. Thomas Kluyver
  13. Thomas Kluyver
  14. Thomas Kluyver
  15. Thomas Kluyver
  16. Thomas Kluyver
  17. Thomas Kluyver
  18. Thomas Kluyver
  19. Thomas Kluyver
  20. Thomas Kluyver
  21. Thomas Kluyver
  22. Thomas Kluyver
  23. Thomas Kluyver
  24. Thomas Kluyver
  25. Thomas Kluyver
  26. Thomas Kluyver

    Fix for nested()

    takluyver authored
  27. Thomas Kluyver
  28. Thomas Kluyver
  29. Thomas Kluyver
  30. Thomas Kluyver
  31. Thomas Kluyver
  32. Thomas Kluyver
  33. Thomas Kluyver
  34. Thomas Kluyver

    Fixes to

    takluyver authored
  35. Thomas Kluyver
  36. Thomas Kluyver
  37. Thomas Kluyver
  38. Thomas Kluyver
  39. Thomas Kluyver

    Fix bugs in PyColorize

    takluyver authored
  40. Thomas Kluyver
  41. Min RK Thomas Kluyver

    add Session.bsession trait for session id as bytes

    minrk authored takluyver committed
  42. Thomas Kluyver
  43. Thomas Kluyver
  44. Grahame Bowland Thomas Kluyver

    Make PNG images in the Qt console work in Python 3.

    grahame authored takluyver committed
Something went wrong with that request. Please try again.