Add python 3 support

[cdeil]

At the moment everyone uses python 2, but it would be nice to support python 3 at some point.

matplotlib 1.2 will be released soon, and then the full scientific python stack will support python 3 and as I understand it rootpy already does, so it would be nice to have python 3 support in rootpy in the mid-term future.

[cdeil]

Indeed rootpy does support python 3.2 already, only a small hack is needed to configure the build with python3:
https://sft.its.cern.ch/jira/browse/ROOT-4627

I've started a python3 branch for rootpy.

[arogozhnikov]

Hi, is there some advance in supporting python3?
The version in PyPI still can't be installed via pip:

    print __doc__
                ^
SyntaxError: invalid syntax

[ndawe]

Not yet, no. I need to catch up with py3 support on the PyROOT side first to fully understand what needs to change (it may be more than just syntactical). Help is welcome.

[sanderbreur]

This would be a great upgrade, we are using Python 3.4 and we need to use Root for the output files. What kind of help would you need? (tasks and expertise?)

[ndawe]

I suggest something like this:

1. Set up a working py3+ROOT build and run the rootpy unit tests to see what fails.
2. Take care of the easy stuff like syntax changes. Probably use https://pypi.python.org/pypi/six to make the compatibility between 2 and 3 easier to handle. The six python file can be placed in rootpy/extern.
3. After working out any needed changes, add the py3+ROOT build to the integration testing system on travis CI to maintain py3 support.

[ghost]

Hey, I managed to run pyROOT with python3.4, what will change now if I consider rootpy?

[ndawe]

Which ROOT version? I just tried v5-34-26 instead of v6 to debug py3 and ROOT6 issues separately. There seem to be lingering py3 incompatibilities in v5-34-26 (py3 syntax errors in genreflex and PyCintex). Ignoring that, this commit here:

https://root.cern.ch/gitweb?p=root.git;a=commitdiff;h=f78843e109

introduced an import hook that doesn't play nice:

Python 3.4.2 (default, Oct  8 2014, 13:08:17) 
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> import setuptools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/home/endw/.local/lib/python3.4/site-packages/setuptools-12.2-py3.4.egg/setuptools/__init__.py", line 4, in <module>
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/distutils/core.py", line 16, in <module>
    from distutils.dist import Distribution
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/distutils/dist.py", line 17, in <module>
    from distutils.util import check_environ, strtobool, rfc822_escape
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/distutils/util.py", line 9, in <module>
    import importlib.util
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/importlib/util.py", line 3, in <module>
    from ._bootstrap import MAGIC_NUMBER
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
ImportError: No module named '_bootstrap'

[cdeil]

I think it's been almost a year, but now I need ROOT again. :-)

My Macports install has ROOT 5.34.30 with Python 3.4 and this works just fine with setuptools 16.0:

import ROOT
import setuptools

There's trivial stuff in rootpy that prevents it from working in Python 3:

$ python setup.py install --user
  File "setup.py", line 93
    print __doc__
                ^
SyntaxError: Missing parentheses in call to 'print'

and if I fix this I get

$ python setup.py install --user
Traceback (most recent call last):
  File "setup.py", line 91, in <module>
    execfile('rootpy/info.py')
NameError: name 'execfile' is not defined

@ndawe or anyone – do you have time to look again into Python 3 support for rootpy? Or should I put it on my TODO list?

[ndawe]

Now that root_numpy is under control, I'm switching my attention back to rootpy (I recently added ROOT 6 and python 3 support for root_numpy and rewrote a lot of the internals for improved speed). I expect that most of the python 3 incompatibilities in rootpy are very trivial. Help is welcome though!

I'm working on this in my spare time over the coming days...

[cdeil]

@ndawe – Sounds good. I'm free next weekend and could work on this (although if complicated ROOT-specific Python 3 issues come up I doubt I'll be able to work them out).

[ndawe]

I made some progress yesterday on this. Now working on the logger magic. Basic PyObject structs changed slightly in python 3 so our dirty injection of opcodes in function frames doesn't work yet.

[ndawe]

@cdeil we also need to work around this:

https://github.com/root-mirror/root/blob/master/bindings/pyroot/src/Converters.cxx#L238

[ndawe]

I think Wim could be using a PyLongObject instead:

https://docs.python.org/3/c-api/long.html#c.PyLongObject

[ndawe]

Working on python 3 support in #645. Managed to get the logger magic working.

[ndawe]

ROOT objects are not hashable in Python 3 because __hash__ is missing:

https://sft.its.cern.ch/jira/browse/ROOT-7365

Could patch this in here in rootpy, but better off putting that in PyROOT.

[cdeil]

With latest master (including #645) I still get a bunch of errors:
https://gist.github.com/cdeil/8cbc91f5d5b49ae0a309

@ndawe – Does everything work for you with Python 3 already or do you see the same / different issues?

[cdeil]

For me import rootpy now works from Python, but not from IPython
(see here) because of this error in rootpy/logger/magic.py in fix_ipython_startup:

AttributeError: 'function' object has no attribute 'im_func'

[ndawe]

ipython issue fixed in ed6de76

[ndawe]

What errors do you have now after #647 ?

[cdeil]

Now I get this: https://gist.github.com/cdeil/20268de5fca2defaa330

[cdeil]

And for the examples: https://gist.github.com/cdeil/582fda0f7894d364ca9c#file-gistfile1-txt-L50

[ndawe]

Ah, I see a few obvious things I missed. I'm not seeing all the same failures because I end up with a segfault near the end of the tests on my linux box.

[ndawe]

The aclic warnings about const seems to be specific to your machine. That's annoying.

[ndawe]

I also see the byteplay3 error: AttributeError: 'str' object has no attribute 'co_code'. Haven't figure that one out.

[ndawe]

Reinstall rootpy before running the examples again. AttributeError: 'function' object has no attribute 'im_func' should no longer happen.

[ndawe]

I would like to remove the dependency on byteplay if possible.

[ndawe]

More fixes went in with #648. Hopefully it's just the byteplay3 error remaining?

[cdeil]

Now I get this for make: https://gist.github.com/cdeil/76321a3a04ef24fb6658
And for make test-examples after re-installing: https://gist.github.com/cdeil/c8d1790ca436daf00642

[ndawe]

Right, the csv test. I need to update the comparison for python 3. You see quite a bit of junk from ROOT.py.

[remenska]

Hi,
I'm working with the XENON1T experiment, and we want to use rootpy with our python 3 code and ROOT6 (although we do care about the rest of the build matrix...e.g, python3/ROOT5, but has lower priority).
We've been looking into what's left for making it work with python 3. There don't seem to be many issues in rootpy, but we thought we'd share some of the "insights" and "fixes" we've done.

• test_attrs was not working with python3.4/ROOT6,
  remenska@bfb9495
• these tests were failing with python3 (Travis-specific, it cannot find the Python3 headers, by default only 2.7 headers are provided). The spitted error was "ERROR: Failure: FileNotFoundError ([Errno 2] No such file or directory: 'pkg-config python --variable=includedir')"
  rootpy.memory.tests.test_keepalive.test_keepalive ... ok
  rootpy.memory.tests.test_keepalive.test_nokeepalive ... ok
  rootpy.memory.tests.test_keepalive.test_canvas_divide ... ok
  remenska@95665c0
• test_csv update
  Added a new root csv file ..the str rounding is different python2/3 as far as I understood (tree.csv function)
  remenska@44a42cf
• as far as the byteplay3 external library goes, I've only tracked down the problem, but haven't solved it.
  There is a slight difference in the bytecode as far as closures go, but I'm really not an expert. This may just give you a hint. Compare:
  python 2: (In [159] is the interesting part)
  https://github.com/remenska/IPythonNotebooks/blob/master/bytecodetest.ipynb
  python 3: (In [103)
  https://github.com/remenska/IPythonNotebooks/blob/master/bytecode3test.ipynb
  notice the extra
  LOAD_CONST 3 ('wrap..outer')
  which is basically a string that makes byteplay3 fail.

Do you maybe have any advice on how to proceed? We're mainly interested in the ROOT I/O, so we're open for help for general python 3 support, if we can.

There were also some small changes just to get it to build with Travis.

Best regards,
Daniela & @tunnell (https://github.com/XENON1T)

[ndawe]

@remenska and @tunnell thank you so much. This is a huge help.

You're right that we aren't too far from full Python 3 support. Just a few annoying details remain.

Would you like to open a pull request for the progress you've made? Even if issues still remain (byteplay...) I can still merge what you've done already.

[remenska]

@ndawe thanks for keeping rootpy in a good condition!
I will cherry-pick the commits and open a pull request.

[parnmatt]

Hi all, I'm unsure if I can be of any use, however I would love to help make a pythonic ROOT a reality with Python 3.

Is there a list of things that need to be done?

[remenska]

Hi @parnmatt,

I will take the liberty of jumping in, as it's also curcial for us to have rootpy work flawlesly with Python 3 :)
You can see the Travis builds here:
https://travis-ci.org/remenska/rootpy

If you focus on Python3/ROOT6 in the build matrix, these are main issues:
rootpy.utils.tests.test_hook.test_inject
The insight I have, from debugging so far, is explained above in my comment (byteplay3/bytecode bullet).

Another problem that needs attention is with c++ STL templates not working with ROOT6 (both with python 2 and 3):
rootpy.tests.test_stl.test_stl
For details, download the log from: https://travis-ci.org/remenska/rootpy/jobs/72454403

There are also skipped tests because Python3 support is still misssing:
rootpy.tree.tests.test_tree.test_chain_draw ... SKIP: Python 3 support not implemented rootpy.tree.tests.test_tree.test_chain_draw_hist_init_first ... SKIP: Python 3 support not implemented

If you could help us out with any of that, I think it will be very appreciated. Let me know if you need some help. Thanks upfront!

Cheers

[cdeil]

After a few years of absence I now have to open ROOT files again.
:-(

So has there been any progress on Python 3 support for ROOT / rootpy?
Last comment in this thread is from Aug 2015 and http://www.rootpy.org/install.html still says Python 3 isn't supported and travis-ci builds fail.

I quickly tried installing via Macports port install root5 +python35, but that gave crashes when I tried to read a histogram from a file. Now trying port install root5 +python27 ...

[tunnell]

At least the Anaconda one works on Python3 since that's what we use within XENON for pyROOT and rootpy. The version could be updated, but @remenska wrote a detailed procedure for this.

[cdeil]

@tunnell - In that case, I think it would be great to update the install page to mention this working conda / Python 3 option. (I just left basically the same comment on this old, closed issue: #642 (comment)).

[ndawe]

I'm only aware of rootpy's logger "magic" not fully working in python3. This was expected since it depends on modifying bytecode in a way that only worked in python2. Otherwise I've hit loads of various PyROOT issues along the way but I think the latest ROOT6 releases are mostly working with python3 now.

Keep in mind that all of this will change with efforts around scikit-hep ramping up and parts of rootpy moving over there. They will be taking over the conda channel too, with updated instructions. We're also exploring the possibility of pip installing ROOT. I can put a note in the rootpy docs about the conda install.

[ndawe]

Python 3 support is now essentially here with the latest tests passing. Remaining python 3 issues can be addressed in separate issues, so closing this now (finally!).

[heroxbd]

Hi @ndawe , thank you!

Should the documentation be updated accordingly?