tf2_py cannot work with python3

[randoms]

compile tf2_py with python3 , cannot import tf2_py

import tf2_py

ImportError: No module named '_tf2'

[tfoote]

How did you compile and setup your environment? There have been PRs recently merged that have been tested using python3. Such as #173

[randoms]

I used virtualenv

source my_work_space
virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
catkin_make

python packages were installed in my_workspace/devel/lib/python3/dist-packages/
start a ipython console
By defaullt /devel/lib/python3/dist-packages/ is not in sys.path, so I added it to sys.path.

[randoms]

If I cd to tf2_py install path and import _tf2, I got the following error

ImportError: dynamic module does not define init function (PyInit__tf2)

tf2_py/src/tf2_py.cpp#L635
defined PyInit_tf2, should that be PyInit__tf2?

[randoms]

I found a way to make it work with python3
tf2_py/src/tf2_py.cpp#L635
from

PyMODINIT_FUNC PyInit_tf2()

to

PyMODINIT_FUNC PyInit__tf2()

tf2_py/src/tf2_py/init.py#L37
from

from _tf2 import *

to

from ._tf2 import *

[tfoote]

Reading through the documentation it does look like the extra underscore is needed: http://python3porting.com/cextensions.html

@de-vri-es @spaghetti- Did you run into this issue?

[de-vri-es]

Indeed, the extra underscore seems to be required. Note that I'm really only compiling code, not running any runtime tests (they won't run for other reasons, although that got fixed recently I think). Shame it doesn't trigger a linker error though.

I don't think the relative import with a dot works in python2 without from __future__ import absolute_import.

[tfoote]

@randoms could you open a PR with your fixes plus the future import @de-vri-es mentioned?

[randoms]

OK

[jacknlliu]

Is this version released for ROS kinetic with Ubuntu 16.04? I still get the error with latest version.

I clone this repo, and using catkin_make build it, with python2, everything works! but with python3 I still get the following error,

$ python3

>>> from tf import TransformListener
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/tf/__init__.py", line 28, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/home/ros/data/tf2_ros/catkin_ws/devel/lib/python2.7/dist-packages/tf2_ros/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/home/ros/data/tf2_ros/catkin_ws/devel/lib/python2.7/dist-packages/tf2_py/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
ImportError: dynamic module does not define module export function (PyInit__tf2)

[jacknlliu]

Could someone reopen this issue? I confirm tf2_py still not work with python3.

[randoms]

From your error message, I think you are loading the python 2.7 version of tf2. To use the python 3 version, you have to compile the source code with python 3. And be careful with your sys.path, python 3 library path should before python 2 library path, make sure to load the right version.

[jacknlliu]

@randoms just using catkin_make did not compile with python3? Doesn't ros package only have the python2 path?

[randoms]

catkin_make uses system python by default.

To use python 3 you need to change your python executable path. As I mentioned in my post before, I used virtualenv. (You may also need to delete your build directory before running catkin_make)

After catkin_make, you can find python 3 libraries in the following path

[jacknlliu]

@randoms thanks very much for your detailed instruction. I tried it, but I still got the following error.

(test)
>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/tf/__init__.py", line 28, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_ros/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_py/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
ImportError: dynamic module does not define module export function (PyInit__tf2)

[randoms]

You need to be careful with your sys.path, python 3 libraries should come before python 2 libraries.

[tfoote]

@randoms please copy and paste the text output instead of screenshotting. Screenshots cannot be searched nor copied from to reproduce.

[jacknlliu]

@randoms I can't reproduce your result.

Here I remove the ros python dist-package path from sys.path, I got the following error.

>>> print(sys.path)
['/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src', 
'/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src', 
'', '/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages',
 '/data/tf2_ros/catkin_ws/test/lib/python35.zip',
 '/data/tf2_ros/catkin_ws/test/lib/python3.5', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/plat-x86_64-linux-gnu', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/lib-dynload',
 '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/site-packages']
>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'tf'

Then I append the ros python package path to the end of sys.path, I got

>>> sys.path.append('/opt/ros/kinetic/lib/python2.7/dist-packages')
>>> print(sys.path)
['/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src', 
'/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src', '',
 '/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages', 
'/data/tf2_ros/catkin_ws/test/lib/python35.zip', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/plat-x86_64-linux-gnu', '/data/tf2_ros/catkin_ws/test/lib/python3.5/lib-dynload', 
'/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/data/tf2_ros/catkin_ws/test/lib/python3.5/site-packages', 
'/opt/ros/kinetic/lib/python2.7/dist-packages']
>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/tf/__init__.py", line 28, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src/tf2_ros/__init__.py", line 38, in <module>
    from tf2_py import *
  File "/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src/tf2_py/__init__.py", line 38, in <module>
    from ._tf2 import *
ImportError: No module named 'tf2_py._tf2'

[randoms]

Let's start from scratch

mkdir catkin_ws
cd catkin_ws
mkdir src
cd src 
git clone https://github.com/ros/geometry
git clone https://github.com/ros/geometry2
cd ..
virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
pip install catkin_pkg pyyaml empy rospkg numpy
catkin_make
source devel/setup.bash

Then run python, and import tf

Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tf
>>> tf.__file__
'/home/xiaoqiang/Desktop/catkin_ws/devel/lib/python3/dist-packages/tf/__init__.py'
>>> import tf2_py
>>> tf2_py.__file__
'/home/xiaoqiang/Desktop/catkin_ws/devel/lib/python3/dist-packages/tf2_py/__init__.py'
>>> 

[jacknlliu]

@randoms I tried it, but I got

>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 30, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_ros/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_py/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
ImportError: dynamic module does not define module export function (PyInit__tf2)
>>> import sys
>>> print(sys.path)
['/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src', 
'/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src', 
'/data/tf2_ros/catkin_ws/src/geometry/tf/src', '', 
'/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages', 
'/data/assembler/assembler-dynamic/devel/lib/python2.7/dist-packages', '/opt/ros/kinetic/lib/python2.7/dist-packages', 
'/data/tf2_ros/catkin_ws/venv/lib/python35.zip',
 '/data/tf2_ros/catkin_ws/venv/lib/python3.5', 
'/data/tf2_ros/catkin_ws/venv/lib/python3.5/plat-x86_64-linux-gnu', '/data/tf2_ros/catkin_ws/venv/lib/python3.5/lib-dynload',
 '/usr/lib/python3.5', 
'/usr/lib/python3.5/plat-x86_64-linux-gnu', 
'/data/tf2_ros/catkin_ws/venv/lib/python3.5/site-packages']

[jacknlliu]

@tfoote What do you think about this? I test this, but I got different output which indicates the incompatibility.

[tfoote]

@jacknlliu You still have python 2.7 in your path. If that's there you're likely getting the wrong versions of libraries. You need to make sure that all elements are built using only python3. (Note that this means no binary packages from debians.)

[jacknlliu]

@tfoote Thanks very much, I will test it. But why not release python3 package to /opt/ros/kinetic/lib/python3/dist-packages, I think this will be easy to use.

[tfoote]

It certainly would be easier to use. For any given package that wouldn't be too much but to do that system wide would require all python dependencies be fully duplicated between python2 and python3 and then both dependencies added, and then full parallel builds for each side of the chain setup. We're looking at doing that in future versions of ROS but in general the overhead for supporting two dual pipelines significantly increases the amount of maintenance required.

[gbanusi]

@jacknlliu I had the same problem on Ubuntu 16.04, and like @tfoote said you need to "get rid" of the python2 packages.

When you execute catkin_make with python3 virtual environment you get this at one point:

and you see it is python2.7 -> you need Python3 header files, so execute this line:

sudo apt-get install python3-dev.

After that you will have sth. like this:

and then everything works like @randoms said.
Btw. thanks @randoms your helped me alot. :)

[tfoote]

@gbanusi It's great you've got it working. Thanks for the followup so others won't hit the same issue. One side note please copy and paste console output instead of screenshotting. That way it's searchable and copyable again (say if I want to grep or run find to help you debug I don't have to retype the text).

@jacknlliu FYI there's a bigger discussion of targeting python version here: ros-infrastructure/rep#154

[Kaju-Bubanja]

@tfoote What exactly is meant by "You need to make sure that all elements are built using only python3."? I found this page, but nowhere do I see an option relating python2 or 3. How would one make sure that all elements are built using only python3?

[gbanusi]

Just make a virtual environment for python3 and activate it as @randoms said,
git clone all the packages (tf2, geometry.... you can find them on ros github) you need to work with in catkin_ws/src (also the ones that are dependent, you know that by the error message),
catkin_make --force-cmake will compile everything with found python3 in your virt. env.,
then everything should work....

[Jendker]

Just in case someone runs into it with ROS Melodic on Ubuntu 18.04, because presented solution which worked fine on 16.04 does not work anymore. I managed to get around it with:

catkin_make -DPYTHON_EXECUTABLE:FILEPATH=/path/to/virtualenv/python

or in case that you are using catkin build:

catkin build --cmake-args -DPYTHON_EXECUTABLE:FILEPATH=/path/to/virtualenv/python

[robodhruv]

I tried following all steps above, but still seem to be getting the same issue. Even while using virtualenv. Any pointers on how to resolve? Do I need to reinstall ROS with some flags?

[Jendker]

The solution from the answer here: https://answers.ros.org/question/326226/importerror-dynamic-module-does-not-define-module-export-function-pyinit__tf2/ seems to still work. As soon as we get ROS Noetic the problem will be completely solved though.

[tfoote]

Do I need to reinstall ROS with some flags?

You cannot use a preinstalled version of ROS. You have to compile it from source with Python3 in as your default python implementation. If you are a power user you can take shortcuts that I won't describe here. If you don't know what they are don't try them.