Add python3 support #196

jcfr · 2017-02-14T14:27:22Z

This set of changes supersedes #195.

It includes changes required to run pyradiomics on Python 3. To gracefully handles the different between py2 and py3, it introduces the dependency on six. See https://pythonhosted.org/six/

Notes:

instead of switching to using dict.items() (which is less efficient in py2), it uses six.iteritems
since MacOSX SimpleITK python wheels are not (yet) available for Python 3.x, corresponding TravisCI builds haven't been enabled.

…2 and py3 Six provides simple utilities for wrapping over differences between Python 2 and Python 3. It is intended to support codebases that work on both Python 2 and 3 without modification. six consists of only one Python file, so it is painless to copy into a project. See https://pythonhosted.org/six/

six.string_types represents the possible types for text data. This is basestring() in Python 2 and str in Python 3.

This was done with the help of 2to3 refactoring tool and the addition of "from __future__ import print_function" where appropriate: $ 2to3 -f print . # dry-run $ 2to3 -f print -w . # proceed to changes

From six documentation [1]: Get the next item of iterator it. StopIteration is raised if the iterator is exhausted. This is a replacement for calling it.next() in Python 2 and next(it) in Python 3. [1] https://pythonhosted.org/six/#object-model-compatibility

From six documentation [1]: Returns an iterator over dictionary‘s items. This replaces dictionary.iteritems() on Python 2 and dictionary.items() on Python 3. kwargs are passed through to the underlying method. [1] https://pythonhosted.org/six/#object-model-compatibility

See https://pythonhosted.org/six/#module-six.moves

…ion. The "cmp" builtin function has been removed from py3 series. See [1] [1] https://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons

To facilitate the update of the notebooks, a convenient script named "fix_notebook_print.py" has been used. Then, import of print_function was manually added. $ ack "print " -l bin/Notebooks/helloRadiomics.ipynb bin/Notebooks/helloFeatureClass.ipynb bin/Notebooks/PyRadiomics Example.ipynb $ /tmp/fix_ipython_notebook_print.py ./bin/Notebooks/helloRadiomics.ipynb bin/Notebooks/helloFeatureClass.ipynb bin/Notebooks/PyRadiomics\ Example.ipynb Fixing ./bin/Notebooks/helloRadiomics.ipynb - done (9 lines updated) Fixing bin/Notebooks/helloFeatureClass.ipynb - done (37 lines updated) Fixing bin/Notebooks/PyRadiomics Example.ipynb - done (25 lines updated) Script available here: https://gist.github.com/jcfr/7366dda36f86deaea2d81ffee160bf3f

Update of the code was facilitated by the use of the 2to3 refactoring tool: $ 2to3 -f dict . # dry-un $ 2to3 -f dict -w . # recactor Updates where code was (1) iterating of the keys values or (2) getting the number of keys wasn't updates. Co-authored-by: Joost van Griethuysen <j.v.griethuysen@nki.nl>

This commit fixes the following error reported when running the tests using python3: File "/home/jcfr/Projects/pyradiomics/tests/testUtils.py", line 184, in readBaselineFiles headers = six.next(csvReader) _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

With newer versions of SimpleITK and python, a tuple or list is expected. Direcly passing a numpy array is not supported. This commit fixes error like the following: File "/home/jcfr/Projects/pyradiomics/radiomics/imageoperations.py", line 125, in cropToTumorMask cif.SetLowerBoundaryCropSize(ijkMinBounds) File "/home/jcfr/.virtualenvs/pyradiomics-py3/lib/python3.5/site-packages/SimpleITK/SimpleITK.py", line 21397, in SetLowerBoundaryCropSize return _SimpleITK.CropImageFilter_SetLowerBoundaryCropSize(self, LowerBoundaryCropSize) TypeError: in method 'CropImageFilter_SetLowerBoundaryCropSize', argument 2 of type 'std::vector< unsigned int,std::allocator< unsigned int > > const &' Co-authored-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>

In python 3.5 GLRLM results differed from baseline. This was due to the different implementation of the `filter()` function, which returns an iterator instead of a list. In GLRLM calculation, there where 2 iterations over this iter-object (one for flat-angle check, one for matrix calculation), causing it to break in python 3.5. Fix by merging the 2 iterations into one loop. Additionally, simplify the filter function.

…str" Some functions expect string types (such as in nose-testing), but python 3.5 by default provides unicode strings. This commit casts where necessary. Also change csv.writer mode from 'wb' to 'w', which accepts regular strings.

Since in Python3, zip() statement returns an iterator, this commit use the list() statement to exhaust the iterator and initialize the numpy array. Co-authored-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>

Since no MacOSX SimpleITK wheels for Python 3.x is available, corresponding Travis builds havn't been enabled.

JoostJM · 2017-02-14T16:12:28Z

.circleci-matrix.yml

-#  - MANYLINUX_IMAGE=manylinux-x64 MANYLINUX_PYTHON=cp35-cp35m
-#  - MANYLINUX_IMAGE=manylinux-x86 MANYLINUX_PYTHON=cp35-cp35m
+  - MANYLINUX_IMAGE=manylinux-x64 MANYLINUX_PYTHON=cp33-cp33m
+  - MANYLINUX_IMAGE=manylinux-x86 MANYLINUX_PYTHON=cp33-cp33m


These two lines cause CircleCI to fail, as it tries to install numpy == 1.12.0, which does not support python 3.3 anymore.

Indeed, I am now looking into adding few env. markers

Thinking more about it, as you suggested let's just remove support for Python 3.3. It is now worth the complication

If really needed, we could look into specifying some more complex requirements using Environment Markers [1] [1] https://www.python.org/dev/peps/pep-0496/ This commit avoids to get the following error: Collecting numpy>=1.9.2 (from -r requirements.txt (line 1)) Downloading numpy-1.12.0.zip (4.8MB) 100% |████████████████████████████████| 4.8MB 240kB/s Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-7w8sbj/numpy/setup.py", line 34, in <module> raise RuntimeError("Python version 2.7 or >= 3.4 required.") RuntimeError: Python version 2.7 or >= 3.4 required. Reported-by: Joost van Griethuysen <j.v.griethuysen@nki.nl>

A new commit addressing the concern has been pushed

JoostJM

Additionally, the variable in_py3 can be removed from __init__.py, as this is replaced by six.PY3

JoostJM · 2017-02-14T16:28:56Z

.travis.yml

@@ -19,11 +19,6 @@ matrix:
 #      env:
 #        - PYTHON_VERSION=3.4.5


Doesn't this mean that python 3.x testing for Mac is still disabled?

python 3.x testing for Mac is still disabled?

Indeed. Waiting for SimpleITK wheels, we do not have a choice. See https://itk.org/SimpleITKDoxygen/html/PyDownloadPage.html

There are however conda packages for SimpleITK, would it make sense to get the mac tests working on miniconda/anaconda (https://anaconda.org/simpleitk/simpleitk) instead of standard python?

That would make sense, the idea would be to submit a recipe to conda-forge. With that in place, the package would be available on conda on linux, macosx and windows.

That said, conda-forge recipe are updated only when there is a release of the package. If you would like to setup continuous integration using miniconda, you would have to set it up on your own.

I currently do not have the bandwidth to set miniconda CI up for pyradiomics, in the mean time I would be happy to review PR.

@blowekamp Do we know when macosx wheels will be available for python 35 ?

I was just working on some scripts for the 1.0 release... I'll see if I can run them on the release branch too like the recent x86_64 0.10.0.post3 wheel.

Suggested-by: Joost van Griethuysen <j.v.griethuysen@nki.nl>

JoostJM · 2017-02-14T16:54:40Z

@jcfr, last build of CircleCI failed, python 3.x couldn't find venv in some commandline argument, I suspect this has something to do with the virtual environment. Can you look into it?

jcfr · 2017-02-14T16:56:08Z

python 3.x couldn't find venv in some commandline argument, I suspect this has something to do with the virtual environment. Can you look into it?

I noticed. I will be looking at this later today.

jcfr · 2017-02-14T22:40:32Z

last build of CircleCI failed, python 3.x couldn't find venv in some commandline argument, I suspect
this has something to do with the virtual environment. Can you look into it?

I identified the problem and submitted a fix to upstream setuptools. See pypa/setuptools#971

In the mean time, I will add a workaround.

…ptools issue Since: (1) CircleCI automatically adds a symlink from /home/ubuntu/pyradiomics/venv to /home/ubuntu/virtualenvs/venv-system, (indeed it detects it is a python project) (2) the symlink is indeed invalid when "seen" from within the docker container building the project (because it mount the current folder into the dockcross/manylinux-x64 image), (3) distutils available in python 3.4.6 has an issue causing sdist to fail when it find an invalid symlink (http://bugs.python.org/issue12885) Setuptools is also failing to monkey patch for that version of python. A patch has been submitted upstream: pypa/setuptools#971 To workaround the issue, we simply remove the symbolic link and avoid the problem completely.

jcfr · 2017-02-14T23:47:34Z

Et voila. As soon as, Travis returns green. Will be integrating 🎆

All comments have been addressed. CircleCI is now green :)

jcfr and others added 16 commits February 14, 2017 10:54

ENH: py3: Use "six.string_types" instead of "basestring"

62c511d

six.string_types represents the possible types for text data. This is basestring() in Python 2 and str in Python 3.

ENH: py3: Update code to use "print" function.

d1862d1

This was done with the help of 2to3 refactoring tool and the addition of "from __future__ import print_function" where appropriate: $ 2to3 -f print . # dry-run $ 2to3 -f print -w . # proceed to changes

ENH: py3: Use "six.moves.range()" instead of "xrange()"

3890702

See https://pythonhosted.org/six/#module-six.moves

ENH: py3: Use "except Exception as exc" syntax

f8ab5da

ENH: py3: Directly compare test settings instead of using "cmp" funct…

6317a52

…ion. The "cmp" builtin function has been removed from py3 series. See [1] [1] https://docs.python.org/3.0/whatsnew/3.0.html#ordering-comparisons

BUG: py3: Fix RadiomicsShape._getMaximum2Ddiameter

9f9f9e3

Since in Python3, zip() statement returns an iterator, this commit use the list() statement to exhaust the iterator and initialize the numpy array. Co-authored-by: Jean-Christophe Fillion-Robin <jchris.fillionr@kitware.com>

ci: Enable py3 testing on Appveyor (Windows) and CircleCI (Linux)

346b4fc

Since no MacOSX SimpleITK wheels for Python 3.x is available, corresponding Travis builds havn't been enabled.

JoostJM previously requested changes Feb 14, 2017

View reviewed changes

JoostJM mentioned this pull request Feb 14, 2017

making it python 3 compatible_REVISED #194

Closed

JoostJM force-pushed the add-python3-support branch from 9df015d to 7ea5480 Compare February 14, 2017 16:24

jcfr force-pushed the add-python3-support branch from 7ea5480 to 2a25157 Compare February 14, 2017 16:25

JoostJM previously requested changes Feb 14, 2017

View reviewed changes

STYL: Remove unused and obsolete radiomics.in_py3 variable

41e4791

Suggested-by: Joost van Griethuysen <j.v.griethuysen@nki.nl>

jcfr force-pushed the add-python3-support branch from f38134a to fc3f1b7 Compare February 14, 2017 22:55

jcfr force-pushed the add-python3-support branch from fc3f1b7 to de8ee88 Compare February 14, 2017 22:56

jcfr merged commit 84dc727 into master Feb 15, 2017

jcfr deleted the add-python3-support branch February 15, 2017 01:35

JoostJM added this to Python 2/3 Compatability in PyRadiomics build/installation Feb 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add python3 support #196

Add python3 support #196

jcfr commented Feb 14, 2017 •

edited

JoostJM Feb 14, 2017

jcfr Feb 14, 2017

jcfr Feb 14, 2017

JoostJM left a comment

JoostJM Feb 14, 2017

jcfr Feb 14, 2017

kmader Feb 27, 2017

JoostJM Feb 28, 2017

jcfr Feb 28, 2017

blowekamp Feb 28, 2017

JoostJM commented Feb 14, 2017

jcfr commented Feb 14, 2017

jcfr commented Feb 14, 2017

jcfr commented Feb 14, 2017

Add python3 support #196

Add python3 support #196

Conversation

jcfr commented Feb 14, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoostJM left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JoostJM commented Feb 14, 2017

jcfr commented Feb 14, 2017

jcfr commented Feb 14, 2017

jcfr commented Feb 14, 2017

jcfr commented Feb 14, 2017 •

edited