Python3 support #250

prabhuramachandran · 2015-09-08T10:25:10Z

The aim of this PR is to solicit feedback and merge this early for those who want to cut themselves on the bleeding edge.

This adds experimental Python3 support for Mayavi. To use it one requires a recent version of apptools and envisage (by recent I mean that these PRs should be merged enthought/apptools#54 and enthought/envisage#61 ).

It requires the master branch of pyface.

It will require the latest version of VTK from their git repository (https://gitlab.kitware.com/vtk/vtk) and a fresh build of VTK configured for Python3. I have not bothered with travis CI support at this point due to the complexity of doing that. These changes pass all current 2.7.x tests on travis so should be safe to merge.

Status and known issues

The changes are compatible with Python 2.x and 3.x. I have tested on 2.7.x and 3.4.x.
All the tvtk tests pass.
There are a couple of random failures in the mayavi tests. These may be version specific issues but need more investigation.
The integration tests mostly run fine but blocks on exit (with PyQt) so if you run one of them you need to kill it to run the next. This can be investigated later.
mlab works quite well.
The mayavi2 application itself has a few issues that seem pyface related and will be investigated later, notably the persistence of the workbench perspectives are not pickled/unpickled properly..

These changes are made by simply running 2to3 and manually applying the suggested changes.

This requires a lot of changes to apptools that are not yet available. More fixes are necessary but a basic `mlab.test_plot3d` works.

The old one does not work with Python3.

A few are still failing intermittently and will be fixed later. These require corresponding changes to apptools.

This was causing some of the UI elements to not show due to an error. Now, the basic dialogs work correctly.

Choose the classes to build in a better way so only those derived from vtkObjectBase are wrapped. Fix issues wrapping certain color traits that use 0-255 color ranges instead of floating point values. Fix test errors.

The center was not being explicitly set and this seems to cause test failures.

The tearDown method was raising an exception too soon causing other unrelated tests to fail.

The `image_data` is the input which is modified using the `change_information_filter`. So the `outputs[0]` is what should really be the dataset generated by this source.

This works around erratic errors that are really not testing errors. `get_engine` is smart enough to provide a null engine when the backend is set to test so we use that instead of creating a fresh NullEngine. This works much more reliably.

This caused test failures since the scene was not picklable and could not be restored.

With the new wrapping mechanism in VTK 6.3 for Python 3, `dir(vtk.vtkSomething)` will contain all the methods including those of the parent whether overridden or not. This causes bloat in the wrappers and also errors since some ignored methods are not not ignored by subclasses.

mabl · 2015-10-06T09:59:48Z

While I still have not got this to work, I've run into some small issues

Sorry: TabError: inconsistent use of tabs and spaces in indentation (qt_gradient_editor.py, line 75)
  File "/home/mabl/PhD/Software/virtualenv3.5/lib/python3.5/site-packages/mayavi-4.4.3-py3.5-linux-x86_64.egg/tvtk/pyface/ui/wx/wxVTKRenderWindowInteractor.py", line 48
    True = 1

The call to build the docs fails on

execfile(os.path.join('..', '..', '..', 'mayavi', '__init__.py'), d)

mabl · 2015-10-06T10:04:46Z

When I work around the doc issue, I can install it, but it will not import mlab:

>>> import mayavi
>>> import mayavi.mlab
Traceback (most recent call last):
  File "tvtk_classes/tvtk_helper.py", line 23, in get_module
ImportError: No module named 'tvtk.custom.light'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/mayavi/mayavi/mlab.py", line 27, in <module>
    from mayavi.tools.camera import view, roll, yaw, pitch, move
  File "/tmp/mayavi/mayavi/tools/camera.py", line 25, in <module>
    from .engine_manager import get_engine
  File "/tmp/mayavi/mayavi/tools/engine_manager.py", line 14, in <module>
    from mayavi.core.engine import Engine
  File "/tmp/mayavi/mayavi/core/engine.py", line 28, in <module>
    from mayavi.core.base import Base
  File "/tmp/mayavi/mayavi/core/base.py", line 19, in <module>
    from tvtk.pyface.tvtk_scene import TVTKScene
  File "/tmp/mayavi/tvtk/pyface/tvtk_scene.py", line 27, in <module>
    from tvtk.pyface import light_manager
  File "/tmp/mayavi/tvtk/pyface/light_manager.py", line 130, in <module>
    class CameraLight(HasTraits):
  File "/tmp/mayavi/tvtk/pyface/light_manager.py", line 146, in CameraLight
    source = Instance(tvtk.Light, ())
  File "tvtk_classes/tvtk_helper.py", line 225, in <lambda>
  File "tvtk_classes/tvtk_helper.py", line 36, in get_class
  File "tvtk_classes/tvtk_helper.py", line 28, in get_module
  File "tvtk_classes/light.py", line 10, in <module>
  File "/tmp/mayavi/tvtk/array_handler.py", line 28, in <module>
    from tvtk.array_ext import set_id_type_array
ImportError: No module named 'tvtk.array_ext'

@mabi

Thanks to @mabi for reporting this.

prabhuramachandran · 2015-10-28T12:59:02Z

@mabl -- thanks for picking those issues, I've pushed fixes for them now.

Apptools saves the script as unicode.

mabl · 2016-01-17T12:35:28Z

@prabhuramachandran This works! Thank you so much! A simple

pip install --upgrade git+https://github.com/prabhuramachandran/mayavi.git@feature/python3

was enough (the dependencies were already up-to-date). You have my vote to get that merged.

prabhuramachandran · 2016-01-18T19:47:38Z

@mabl -- thanks for testing this! You should actually use the https://github.com/enthought/mayavi.git repositories feature/python3 branch as it is the main one. Will ask @dmsurti to do a review and merge this ASAP.

dmsurti · 2016-01-19T08:01:35Z

@prabhuramachandran The appveyor failure is due to this known issue, #233.

Needs a docs updated in a separate PR: Issue #274. Also issues need to be opened for known issues listed in the PR description. The other issues need to be opened for: Updating CI for both py2/py3.

My only question is around why we did not use a tool (modernize/futurize) to update the code? But otherwise LGTM.

Once the issues are listed, which I will do later, I will merge this!!!

dmsurti · 2016-01-20T14:48:07Z

@prabhuramachandran All known issues have been logged. Will you be pushing more updates to this PR?

prabhuramachandran · 2016-01-21T06:48:14Z

@dmsurti -- No, nothing planned for now. It would be good to have this merged and then work on the other issues slowly. We could potentially put out a release as I think this resolves a bunch of issues with recent VTK versions.

Merged. Ingoring the appveyor failure as it is due to this known issue, #233.

dmsurti · 2016-01-21T07:17:45Z

LGTM. Thanks @prabhuramachandran.

jeremyherbert · 2016-01-22T07:54:16Z

I currently can't get this to work using the git master:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-c7e202e132f1> in <module>()
----> 1 import mayavi.mlab

/Users/jeremy/anaconda/lib/python3.5/site-packages/mayavi/mlab.py in <module>()
     25
     26 # Mayavi imports
---> 27 from mayavi.tools.camera import view, roll, yaw, pitch, move
     28 from mayavi.tools.figure import figure, clf, gcf, savefig, \
     29     draw, sync_camera, close, screenshot

/Users/jeremy/anaconda/lib/python3.5/site-packages/mayavi/tools/camera.py in <module>()
     23 # We can't use gcf, as it creates a circular import in camera management
     24 # routines.
---> 25 from .engine_manager import get_engine
     26
     27

/Users/jeremy/anaconda/lib/python3.5/site-packages/mayavi/tools/engine_manager.py in <module>()
     10
     11 # Local imports
---> 12 from mayavi.preferences.api import preference_manager
     13 from mayavi.core.registry import registry
     14 from mayavi.core.engine import Engine

/Users/jeremy/anaconda/lib/python3.5/site-packages/mayavi/preferences/api.py in <module>()
      2
      3 # The global PreferenceManager instance
----> 4 from .preference_manager import preference_manager
      5 from .bindings import set_scene_preferences, get_scene_preferences

/Users/jeremy/anaconda/lib/python3.5/site-packages/mayavi/preferences/preference_manager.py in <module>()
     28 from traits.api import HasTraits, Instance
     29 from traitsui.api import View, Group, Item
---> 30 from apptools.preferences.api import (ScopedPreferences, IPreferences,
     31         PreferencesHelper)
     32

/Users/jeremy/anaconda/lib/python3.5/site-packages/apptools/preferences/api.py in <module>()
----> 1 from i_preferences import IPreferences
      2
      3 from package_globals import get_default_preferences, set_default_preferences
      4 from preferences import Preferences
      5 from preference_binding import PreferenceBinding, bind_preference

ImportError: No module named 'i_preferences'

I'm using anaconda python 3.5 on OSX. VTK is built and I can import it successfully. I installed this by downloading the zip of the repo, and running pip install on it. Any help would be appreciated.

jeremyherbert · 2016-01-22T08:04:43Z

Never mind, I worked it out.

For anyone else, assuming you like to live life on the edge, you need to pip install

https://github.com/enthought/apptools/archive/master.zip
https://github.com/enthought/envisage/archive/master.zip

solarjoe · 2016-04-05T08:58:40Z

Just some quick postive feedback
I am using WinPython-64bit-3.4.4.1 with Mayavi 4.4.4 and it works fine. Thanks to all devs!

1.) VTK-7.0.0-cp34-cp34m-win_amd64.whl (http://www.lfd.uci.edu/~gohlke/pythonlibs/#vtk)
2.) pip install mayavi

indranilsinharoy · 2016-04-09T01:23:37Z

@solarjoe
Hi,
I was really excited seeing your feedback, as I have not been able to install Mayavi properly lately.

I installed WinPython-64bit-3.4.4.1, following which I downloaded VTK-7.0.0-cp34-cp34m-win_amd64.whl from the above link. However,, when I try to install VTK using the WinPython control panel, I get the following error message:

The following packages filenaming are not recognized by WinPython Control Panel: VTK-7.0.0-cp34-cp34m-win_amd64.whl

My environment is 64 bit, Windows 7. Did you ever see the above error, or would you know how to resolve this issue? Any suggestions will be useful.

Thank you very much.
Regards,
Indranil.

solarjoe · 2016-04-09T06:20:31Z

Hello @indranilsinharoy,

yes, I also came across that issue, it should be fixed in the next WinPython release.
As a workaround I did what is mentioned here:
winpython/winpython#258

Other workarounds to fix the naming detection are listed here:
winpython/winpython#303
winpython/winpython#260

Or you can rename the package, I guess VTK-7.0.0-cp34-none-win_amd64.whl
should work. But I recommend some of the other suggestions to fix it permanently.

Have fun with Mayavi!

indranilsinharoy · 2016-04-09T20:28:06Z

@solarjoe ,

Thank you very much. The first solution worked out rather well for WinPython-64bit-3.4.4.1.

Interestingly, since I already had WinPython-64bit-3.5.1.1 I was eager to test the solution on WinPython-64bit-3.5 (that want I wouldn't need to keep two version of WinPython). For whatever reason, I couldn't get VTK-7.0.0-cp35-cp35m-win_amd64.whl to install. So, for now I will just stick with WinPython-64bit-3.4.4.1.

prabhuramachandran added 22 commits August 17, 2015 23:00

First cut of Python 3 changes for TVTK.

e44df56

ENH: Make all code Python 3 compliant.

0ba83ea

These changes are made by simply running 2to3 and manually applying the suggested changes.

Changes to get all TVTK tests passing with py3.

a2ff8c1

Mlab can now be imported and used with python3.

c098f94

This requires a lot of changes to apptools that are not yet available. More fixes are necessary but a basic `mlab.test_plot3d` works.

Updating to newer traits adaptation.

a262077

The old one does not work with Python3.

Almost all mayavi tests pass.

8f48871

A few are still failing intermittently and will be fixed later. These require corresponding changes to apptools.

Few more fixes to the core.

ca67cb7

More fixes to the core.

8dad2a6

Changes to integrationtests for Python3.

9047b20

BUG: Missed converting use of "file".

3f9e1f5

This was causing some of the UI elements to not show due to an error. Now, the basic dialogs work correctly.

Fix TVTK issues on Python 2 and 3.

588357e

Choose the classes to build in a better way so only those derived from vtkObjectBase are wrapped. Fix issues wrapping certain color traits that use 0-255 color ranges instead of floating point values. Fix test errors.

Fix auto doc example generation scripts.

14c6b5b

Fix a bug with VTKDataSource and a TVTK test error.

844b83c

Fix another issue with Python2/3 in test.

7daee91

Fix tests on Python2.

05fb026

Fix a couple of tests.

6d96751

The center was not being explicitly set and this seems to cause test failures.

Make failures a bit more reliable.

85b5422

The tearDown method was raising an exception too soon causing other unrelated tests to fail.

BUG: Use the outputs of the source.

dea45c8

The `image_data` is the input which is modified using the `change_information_filter`. So the `outputs[0]` is what should really be the dataset generated by this source.

BUG: The cursor cannot be pickled.

75cdc27

This caused test failures since the scene was not picklable and could not be restored.

Update image testing code to new pipeline.

5966293

jenshnielsen mentioned this pull request Oct 5, 2015

Python 3 support #84

Closed

mwaskom mentioned this pull request Oct 5, 2015

Python 3 support nipy/PySurfer#143

Closed

prabhuramachandran added 2 commits October 28, 2015 18:26

Replace execfile with an exec.

7d2b998

Fix tabnanny issues.

0706f52

Thanks to @mabi for reporting this.

Fix bug in test for Python 2.7.

a811639

Apptools saves the script as unicode.

dmsurti mentioned this pull request Jan 19, 2016

Add python3 support documentation #274

Closed

mwaskom mentioned this pull request Jan 20, 2016

backends and multithreading nipy/PySurfer#106

Closed

This was referenced Jan 20, 2016

Fix mayavi test failures in python3 upgrade #275

Open

Fix integration tests in python3 upgrade #276

Open

Fix mayavi2 pyface related problems in python3 upgrade #277

Open

dmsurti added a commit that referenced this pull request Jan 21, 2016

Merge pull request #250 from enthought/feature/python3

02d6871

Merged. Ingoring the appveyor failure as it is due to this known issue, #233.

dmsurti merged commit 02d6871 into master Jan 21, 2016

francisco-dlp mentioned this pull request Jan 21, 2016

ENH python3 hyperspy/hyperspy#769

Merged

8 tasks

This was referenced Jan 21, 2016

Partial support for python3 sphinx-build #266

Closed

Check for a find method of parsed VTK docstring #269

Closed

dmsurti mentioned this pull request Jan 22, 2016

Mayavi unit test_recorder fails on VTK 6.3 #279

Closed

jenshnielsen mentioned this pull request Feb 3, 2016

problem with install in python 3.4.3 ubuntu/elementary freya 64 bit #282

Closed

dmsurti mentioned this pull request Feb 25, 2016

Update Mayavi to use VTK Version 6.3 #248

Closed

prabhuramachandran mentioned this pull request Mar 2, 2016

SetSize needs to be called as an unbound method (ref: Kitware/VTK) #304

Merged

leouieda mentioned this pull request Jul 14, 2016

Python 3 compatibility fatiando/fatiando#260

Open

mdickinson deleted the feature/python3 branch August 1, 2016 08:38

MichaelClerx mentioned this pull request Feb 23, 2018

Make Myokit with work Python 2 and 3 myokit/myokit#26

Closed

jonathanrocher mentioned this pull request Apr 3, 2018

troubles installing mayavi in Anaconda 3.6 #625

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python3 support #250

Python3 support #250

prabhuramachandran commented Sep 8, 2015

mabl commented Oct 6, 2015

mabl commented Oct 6, 2015

prabhuramachandran commented Oct 28, 2015

mabl commented Jan 17, 2016

prabhuramachandran commented Jan 18, 2016

dmsurti commented Jan 19, 2016

dmsurti commented Jan 20, 2016

prabhuramachandran commented Jan 21, 2016

dmsurti commented Jan 21, 2016

jeremyherbert commented Jan 22, 2016

jeremyherbert commented Jan 22, 2016

solarjoe commented Apr 5, 2016

indranilsinharoy commented Apr 9, 2016

solarjoe commented Apr 9, 2016

indranilsinharoy commented Apr 9, 2016

Python3 support #250

Python3 support #250

Conversation

prabhuramachandran commented Sep 8, 2015

Status and known issues

mabl commented Oct 6, 2015

mabl commented Oct 6, 2015

prabhuramachandran commented Oct 28, 2015

mabl commented Jan 17, 2016

prabhuramachandran commented Jan 18, 2016

dmsurti commented Jan 19, 2016

dmsurti commented Jan 20, 2016

prabhuramachandran commented Jan 21, 2016

dmsurti commented Jan 21, 2016

jeremyherbert commented Jan 22, 2016

jeremyherbert commented Jan 22, 2016

solarjoe commented Apr 5, 2016

indranilsinharoy commented Apr 9, 2016

solarjoe commented Apr 9, 2016

indranilsinharoy commented Apr 9, 2016