Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Add python 3 support #35

Closed
cdeil opened this Issue Sep 9, 2012 · 39 comments

Comments

Projects
None yet
8 participants
Owner

cdeil commented Sep 9, 2012

At the moment everyone uses python 2, but it would be nice to support python 3 at some point.

matplotlib 1.2 will be released soon, and then the full scientific python stack will support python 3 and as I understand it rootpy already does, so it would be nice to have python 3 support in rootpy in the mid-term future.

@cdeil cdeil was assigned Sep 9, 2012

Owner

cdeil commented Nov 5, 2012

Indeed rootpy does support python 3.2 already, only a small hack is needed to configure the build with python3:
https://sft.its.cern.ch/jira/browse/ROOT-4627

I've started a python3 branch for rootpy.

Hi, is there some advance in supporting python3?
The version in PyPI still can't be installed via pip:

    print __doc__
                ^
SyntaxError: invalid syntax
Owner

ndawe commented Nov 19, 2014

Not yet, no. I need to catch up with py3 support on the PyROOT side first to fully understand what needs to change (it may be more than just syntactical). Help is welcome.

This would be a great upgrade, we are using Python 3.4 and we need to use Root for the output files. What kind of help would you need? (tasks and expertise?)

Owner

ndawe commented Nov 25, 2014

I suggest something like this:

  1. Set up a working py3+ROOT build and run the rootpy unit tests to see what fails.
  2. Take care of the easy stuff like syntax changes. Probably use https://pypi.python.org/pypi/six to make the compatibility between 2 and 3 easier to handle. The six python file can be placed in rootpy/extern.
  3. After working out any needed changes, add the py3+ROOT build to the integration testing system on travis CI to maintain py3 support.

vsilv commented Feb 20, 2015

Hey, I managed to run pyROOT with python3.4, what will change now if I consider rootpy?

Owner

ndawe commented Feb 23, 2015

Which ROOT version? I just tried v5-34-26 instead of v6 to debug py3 and ROOT6 issues separately. There seem to be lingering py3 incompatibilities in v5-34-26 (py3 syntax errors in genreflex and PyCintex). Ignoring that, this commit here:

https://root.cern.ch/gitweb?p=root.git;a=commitdiff;h=f78843e109

introduced an import hook that doesn't play nice:

Python 3.4.2 (default, Oct  8 2014, 13:08:17) 
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ROOT
>>> import setuptools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/home/endw/.local/lib/python3.4/site-packages/setuptools-12.2-py3.4.egg/setuptools/__init__.py", line 4, in <module>
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/distutils/core.py", line 16, in <module>
    from distutils.dist import Distribution
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/distutils/dist.py", line 17, in <module>
    from distutils.util import check_environ, strtobool, rfc822_escape
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/distutils/util.py", line 9, in <module>
    import importlib.util
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
  File "/usr/lib/python3.4/importlib/util.py", line 3, in <module>
    from ._bootstrap import MAGIC_NUMBER
  File "/home/endw/software/root/root_v5.34.26_py3/lib/root/ROOT.py", line 354, in _importhook
    return _orig_ihook( name, glbls, lcls, fromlist )
ImportError: No module named '_bootstrap'
Owner

cdeil commented May 25, 2015

I think it's been almost a year, but now I need ROOT again. :-)

My Macports install has ROOT 5.34.30 with Python 3.4 and this works just fine with setuptools 16.0:

import ROOT
import setuptools

There's trivial stuff in rootpy that prevents it from working in Python 3:

$ python setup.py install --user
  File "setup.py", line 93
    print __doc__
                ^
SyntaxError: Missing parentheses in call to 'print'

and if I fix this I get

$ python setup.py install --user
Traceback (most recent call last):
  File "setup.py", line 91, in <module>
    execfile('rootpy/info.py')
NameError: name 'execfile' is not defined

@ndawe or anyone – do you have time to look again into Python 3 support for rootpy? Or should I put it on my TODO list?

Owner

ndawe commented May 26, 2015

Now that root_numpy is under control, I'm switching my attention back to rootpy (I recently added ROOT 6 and python 3 support for root_numpy and rewrote a lot of the internals for improved speed). I expect that most of the python 3 incompatibilities in rootpy are very trivial. Help is welcome though!

I'm working on this in my spare time over the coming days...

Owner

cdeil commented May 26, 2015

@ndawe – Sounds good. I'm free next weekend and could work on this (although if complicated ROOT-specific Python 3 issues come up I doubt I'll be able to work them out).

Owner

ndawe commented May 27, 2015

I made some progress yesterday on this. Now working on the logger magic. Basic PyObject structs changed slightly in python 3 so our dirty injection of opcodes in function frames doesn't work yet.

Owner

ndawe commented May 27, 2015

I think Wim could be using a PyLongObject instead:

https://docs.python.org/3/c-api/long.html#c.PyLongObject

Owner

ndawe commented May 27, 2015

Working on python 3 support in #645. Managed to get the logger magic working.

Owner

ndawe commented May 28, 2015

ROOT objects are not hashable in Python 3 because __hash__ is missing:

https://sft.its.cern.ch/jira/browse/ROOT-7365

Could patch this in here in rootpy, but better off putting that in PyROOT.

Owner

cdeil commented Jun 2, 2015

With latest master (including #645) I still get a bunch of errors:
https://gist.github.com/cdeil/8cbc91f5d5b49ae0a309

@ndawe – Does everything work for you with Python 3 already or do you see the same / different issues?

Owner

cdeil commented Jun 2, 2015

For me import rootpy now works from Python, but not from IPython
(see here) because of this error in rootpy/logger/magic.py in fix_ipython_startup:

AttributeError: 'function' object has no attribute 'im_func'
Owner

ndawe commented Jun 2, 2015

ipython issue fixed in ed6de76

Owner

ndawe commented Jun 2, 2015

What errors do you have now after #647 ?

Owner

ndawe commented Jun 2, 2015

Ah, I see a few obvious things I missed. I'm not seeing all the same failures because I end up with a segfault near the end of the tests on my linux box.

Owner

ndawe commented Jun 2, 2015

The aclic warnings about const seems to be specific to your machine. That's annoying.

Owner

ndawe commented Jun 2, 2015

I also see the byteplay3 error: AttributeError: 'str' object has no attribute 'co_code'. Haven't figure that one out.

Owner

ndawe commented Jun 2, 2015

Reinstall rootpy before running the examples again. AttributeError: 'function' object has no attribute 'im_func' should no longer happen.

Owner

ndawe commented Jun 2, 2015

I would like to remove the dependency on byteplay if possible.

Owner

ndawe commented Jun 2, 2015

More fixes went in with #648. Hopefully it's just the byteplay3 error remaining?

Owner

cdeil commented Jun 2, 2015

Now I get this for make: https://gist.github.com/cdeil/76321a3a04ef24fb6658
And for make test-examples after re-installing: https://gist.github.com/cdeil/c8d1790ca436daf00642

Owner

ndawe commented Jun 2, 2015

Right, the csv test. I need to update the comparison for python 3. You see quite a bit of junk from ROOT.py.

@tunnell tunnell referenced this issue in XENON1T/pax Jun 24, 2015

Closed

Consistency on ROOT file interfaces #176

Contributor

remenska commented Jul 20, 2015

Hi,
I'm working with the XENON1T experiment, and we want to use rootpy with our python 3 code and ROOT6 (although we do care about the rest of the build matrix...e.g, python3/ROOT5, but has lower priority).
We've been looking into what's left for making it work with python 3. There don't seem to be many issues in rootpy, but we thought we'd share some of the "insights" and "fixes" we've done.

  • test_attrs was not working with python3.4/ROOT6,
    remenska/rootpy@bfb9495
  • these tests were failing with python3 (Travis-specific, it cannot find the Python3 headers, by default only 2.7 headers are provided). The spitted error was "ERROR: Failure: FileNotFoundError ([Errno 2] No such file or directory: 'pkg-config python --variable=includedir')"
    rootpy.memory.tests.test_keepalive.test_keepalive ... ok
    rootpy.memory.tests.test_keepalive.test_nokeepalive ... ok
    rootpy.memory.tests.test_keepalive.test_canvas_divide ... ok
    remenska/rootpy@95665c0
  • test_csv update
    Added a new root csv file ..the str rounding is different python2/3 as far as I understood (tree.csv function)
    remenska/rootpy@44a42cf
  • as far as the byteplay3 external library goes, I've only tracked down the problem, but haven't solved it.
    There is a slight difference in the bytecode as far as closures go, but I'm really not an expert. This may just give you a hint. Compare:
    python 2: (In [159] is the interesting part)
    https://github.com/remenska/IPythonNotebooks/blob/master/bytecodetest.ipynb
    python 3: (In [103)
    https://github.com/remenska/IPythonNotebooks/blob/master/bytecode3test.ipynb
    notice the extra
    LOAD_CONST 3 ('wrap..outer')
    which is basically a string that makes byteplay3 fail.

Do you maybe have any advice on how to proceed? We're mainly interested in the ROOT I/O, so we're open for help for general python 3 support, if we can.

There were also some small changes just to get it to build with Travis.

Best regards,
Daniela & @tunnell (https://github.com/XENON1T)

Owner

ndawe commented Jul 21, 2015

@remenska and @tunnell thank you so much. This is a huge help.

You're right that we aren't too far from full Python 3 support. Just a few annoying details remain.

Would you like to open a pull request for the progress you've made? Even if issues still remain (byteplay...) I can still merge what you've done already.

Contributor

remenska commented Jul 22, 2015

@ndawe thanks for keeping rootpy in a good condition!
I will cherry-pick the commits and open a pull request.

Hi all, I'm unsure if I can be of any use, however I would love to help make a pythonic ROOT a reality with Python 3.

Is there a list of things that need to be done?

Contributor

remenska commented Aug 28, 2015

Hi @parnmatt,

I will take the liberty of jumping in, as it's also curcial for us to have rootpy work flawlesly with Python 3 :)
You can see the Travis builds here:
https://travis-ci.org/remenska/rootpy

If you focus on Python3/ROOT6 in the build matrix, these are main issues:
rootpy.utils.tests.test_hook.test_inject
The insight I have, from debugging so far, is explained above in my comment (byteplay3/bytecode bullet).

Another problem that needs attention is with c++ STL templates not working with ROOT6 (both with python 2 and 3):
rootpy.tests.test_stl.test_stl
For details, download the log from: https://travis-ci.org/remenska/rootpy/jobs/72454403

There are also skipped tests because Python3 support is still misssing:
rootpy.tree.tests.test_tree.test_chain_draw ... SKIP: Python 3 support not implemented rootpy.tree.tests.test_tree.test_chain_draw_hist_init_first ... SKIP: Python 3 support not implemented

If you could help us out with any of that, I think it will be very appreciated. Let me know if you need some help. Thanks upfront!

Cheers

Owner

cdeil commented Jan 19, 2017

After a few years of absence I now have to open ROOT files again.
:-(

So has there been any progress on Python 3 support for ROOT / rootpy?
Last comment in this thread is from Aug 2015 and http://www.rootpy.org/install.html still says Python 3 isn't supported and travis-ci builds fail.

I quickly tried installing via Macports port install root5 +python35, but that gave crashes when I tried to read a histogram from a file. Now trying port install root5 +python27 ...

tunnell commented Jan 19, 2017

At least the Anaconda one works on Python3 since that's what we use within XENON for pyROOT and rootpy. The version could be updated, but @remenska wrote a detailed procedure for this.

Owner

cdeil commented Jan 19, 2017

@tunnell - In that case, I think it would be great to update the install page to mention this working conda / Python 3 option. (I just left basically the same comment on this old, closed issue: #642 (comment)).

Owner

ndawe commented Jan 20, 2017

I'm only aware of rootpy's logger "magic" not fully working in python3. This was expected since it depends on modifying bytecode in a way that only worked in python2. Otherwise I've hit loads of various PyROOT issues along the way but I think the latest ROOT6 releases are mostly working with python3 now.

Keep in mind that all of this will change with efforts around scikit-hep ramping up and parts of rootpy moving over there. They will be taking over the conda channel too, with updated instructions. We're also exploring the possibility of pip installing ROOT. I can put a note in the rootpy docs about the conda install.

Owner

ndawe commented Jul 7, 2017

Python 3 support is now essentially here with the latest tests passing. Remaining python 3 issues can be addressed in separate issues, so closing this now (finally!).

@ndawe ndawe closed this Jul 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment