New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tf2_py cannot work with python3 #259

Closed
randoms opened this Issue Aug 8, 2017 · 27 comments

Comments

Projects
None yet
6 participants
@randoms
Contributor

randoms commented Aug 8, 2017

compile tf2_py with python3 , cannot import tf2_py

import tf2_py

ImportError: No module named '_tf2'
@tfoote

This comment has been minimized.

Member

tfoote commented Aug 8, 2017

How did you compile and setup your environment? There have been PRs recently merged that have been tested using python3. Such as #173

@randoms

This comment has been minimized.

Contributor

randoms commented Aug 8, 2017

I used virtualenv

source my_work_space
virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
catkin_make

python packages were installed in my_workspace/devel/lib/python3/dist-packages/
start a ipython console
By defaullt /devel/lib/python3/dist-packages/ is not in sys.path, so I added it to sys.path.

image

@randoms

This comment has been minimized.

Contributor

randoms commented Aug 8, 2017

If I cd to tf2_py install path and import _tf2, I got the following error

ImportError: dynamic module does not define init function (PyInit__tf2)

tf2_py/src/tf2_py.cpp#L635
defined PyInit_tf2, should that be PyInit__tf2?

@randoms

This comment has been minimized.

Contributor

randoms commented Aug 8, 2017

I found a way to make it work with python3
tf2_py/src/tf2_py.cpp#L635
from

PyMODINIT_FUNC PyInit_tf2()

to

PyMODINIT_FUNC PyInit__tf2()

tf2_py/src/tf2_py/init.py#L37
from

from _tf2 import *

to

from ._tf2 import *
@tfoote

This comment has been minimized.

Member

tfoote commented Aug 8, 2017

Reading through the documentation it does look like the extra underscore is needed: http://python3porting.com/cextensions.html

@de-vri-es @spaghetti- Did you run into this issue?

@de-vri-es

This comment has been minimized.

Contributor

de-vri-es commented Aug 8, 2017

Indeed, the extra underscore seems to be required. Note that I'm really only compiling code, not running any runtime tests (they won't run for other reasons, although that got fixed recently I think). Shame it doesn't trigger a linker error though.

I don't think the relative import with a dot works in python2 without from __future__ import absolute_import.

@tfoote

This comment has been minimized.

Member

tfoote commented Aug 8, 2017

@randoms could you open a PR with your fixes plus the future import @de-vri-es mentioned?

@randoms

This comment has been minimized.

Contributor

randoms commented Aug 9, 2017

OK

@randoms randoms closed this Aug 9, 2017

@jacknlliu

This comment has been minimized.

jacknlliu commented Dec 3, 2017

Is this version released for ROS kinetic with Ubuntu 16.04? I still get the error with latest version.

I clone this repo, and using catkin_make build it, with python2, everything works! but with python3 I still get the following error,

$ python3

>>> from tf import TransformListener
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/tf/__init__.py", line 28, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/home/ros/data/tf2_ros/catkin_ws/devel/lib/python2.7/dist-packages/tf2_ros/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/home/ros/data/tf2_ros/catkin_ws/devel/lib/python2.7/dist-packages/tf2_py/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
ImportError: dynamic module does not define module export function (PyInit__tf2)
@jacknlliu

This comment has been minimized.

jacknlliu commented Dec 20, 2017

Could someone reopen this issue? I confirm tf2_py still not work with python3.

@randoms

This comment has been minimized.

Contributor

randoms commented Dec 20, 2017

From your error message, I think you are loading the python 2.7 version of tf2. To use the python 3 version, you have to compile the source code with python 3. And be careful with your sys.path, python 3 library path should before python 2 library path, make sure to load the right version.

@jacknlliu

This comment has been minimized.

jacknlliu commented Dec 20, 2017

@randoms just using catkin_make did not compile with python3? Doesn't ros package only have the python2 path?

@randoms

This comment has been minimized.

Contributor

randoms commented Dec 21, 2017

catkin_make uses system python by default.

image

To use python 3 you need to change your python executable path. As I mentioned in my post before, I used virtualenv. (You may also need to delete your build directory before running catkin_make)

image

After catkin_make, you can find python 3 libraries in the following path

image

@jacknlliu

This comment has been minimized.

jacknlliu commented Dec 21, 2017

@randoms thanks very much for your detailed instruction. I tried it, but I still got the following error.

(test)
>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/tf/__init__.py", line 28, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_ros/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_py/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
ImportError: dynamic module does not define module export function (PyInit__tf2)
@randoms

This comment has been minimized.

Contributor

randoms commented Dec 21, 2017

You need to be careful with your sys.path, python 3 libraries should come before python 2 libraries.

image

@tfoote

This comment has been minimized.

Member

tfoote commented Dec 21, 2017

@randoms please copy and paste the text output instead of screenshotting. Screenshots cannot be searched nor copied from to reproduce.

@jacknlliu

This comment has been minimized.

jacknlliu commented Dec 21, 2017

@randoms I can't reproduce your result.

Here I remove the ros python dist-package path from sys.path, I got the following error.

>>> print(sys.path)
['/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src', 
'/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src', 
'', '/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages',
 '/data/tf2_ros/catkin_ws/test/lib/python35.zip',
 '/data/tf2_ros/catkin_ws/test/lib/python3.5', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/plat-x86_64-linux-gnu', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/lib-dynload',
 '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/site-packages']
>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'tf'

Then I append the ros python package path to the end of sys.path, I got

>>> sys.path.append('/opt/ros/kinetic/lib/python2.7/dist-packages')
>>> print(sys.path)
['/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src', 
'/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src', '',
 '/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages', 
'/data/tf2_ros/catkin_ws/test/lib/python35.zip', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5', 
'/data/tf2_ros/catkin_ws/test/lib/python3.5/plat-x86_64-linux-gnu', '/data/tf2_ros/catkin_ws/test/lib/python3.5/lib-dynload', 
'/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/data/tf2_ros/catkin_ws/test/lib/python3.5/site-packages', 
'/opt/ros/kinetic/lib/python2.7/dist-packages']
>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/tf/__init__.py", line 28, in <module>
    from tf2_ros import TransformException as Exception, ConnectivityException, LookupException, ExtrapolationException
  File "/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src/tf2_ros/__init__.py", line 38, in <module>
    from tf2_py import *
  File "/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src/tf2_py/__init__.py", line 38, in <module>
    from ._tf2 import *
ImportError: No module named 'tf2_py._tf2'
@randoms

This comment has been minimized.

Contributor

randoms commented Dec 21, 2017

Let's start from scratch

mkdir catkin_ws
cd catkin_ws
mkdir src
cd src 
git clone https://github.com/ros/geometry
git clone https://github.com/ros/geometry2
cd ..
virtualenv -p /usr/bin/python3 venv
source venv/bin/activate
pip install catkin_pkg pyyaml empy rospkg numpy
catkin_make
source devel/setup.bash

Then run python, and import tf

Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tf
>>> tf.__file__
'/home/xiaoqiang/Desktop/catkin_ws/devel/lib/python3/dist-packages/tf/__init__.py'
>>> import tf2_py
>>> tf2_py.__file__
'/home/xiaoqiang/Desktop/catkin_ws/devel/lib/python3/dist-packages/tf2_py/__init__.py'
>>> 
@jacknlliu

This comment has been minimized.

jacknlliu commented Dec 21, 2017

@randoms I tried it, but I got

>>> import tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 30, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_ros/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
  File "/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages/tf2_py/__init__.py", line 35, in <module>
    exec(__fh.read())
  File "<string>", line 38, in <module>
ImportError: dynamic module does not define module export function (PyInit__tf2)
>>> import sys
>>> print(sys.path)
['/data/tf2_ros/catkin_ws/src/geometry2/tf2_py/src', 
'/data/tf2_ros/catkin_ws/src/geometry2/tf2_ros/src', 
'/data/tf2_ros/catkin_ws/src/geometry/tf/src', '', 
'/data/tf2_ros/catkin_ws/devel/lib/python3/dist-packages', 
'/data/assembler/assembler-dynamic/devel/lib/python2.7/dist-packages', '/opt/ros/kinetic/lib/python2.7/dist-packages', 
'/data/tf2_ros/catkin_ws/venv/lib/python35.zip',
 '/data/tf2_ros/catkin_ws/venv/lib/python3.5', 
'/data/tf2_ros/catkin_ws/venv/lib/python3.5/plat-x86_64-linux-gnu', '/data/tf2_ros/catkin_ws/venv/lib/python3.5/lib-dynload',
 '/usr/lib/python3.5', 
'/usr/lib/python3.5/plat-x86_64-linux-gnu', 
'/data/tf2_ros/catkin_ws/venv/lib/python3.5/site-packages']
@jacknlliu

This comment has been minimized.

jacknlliu commented Jan 24, 2018

@tfoote What do you think about this? I test this, but I got different output which indicates the incompatibility.

@tfoote

This comment has been minimized.

Member

tfoote commented Jan 24, 2018

@jacknlliu You still have python 2.7 in your path. If that's there you're likely getting the wrong versions of libraries. You need to make sure that all elements are built using only python3. (Note that this means no binary packages from debians.)

@jacknlliu

This comment has been minimized.

jacknlliu commented Jan 25, 2018

@tfoote Thanks very much, I will test it. But why not release python3 package to /opt/ros/kinetic/lib/python3/dist-packages, I think this will be easy to use.

@tfoote

This comment has been minimized.

Member

tfoote commented Jan 25, 2018

It certainly would be easier to use. For any given package that wouldn't be too much but to do that system wide would require all python dependencies be fully duplicated between python2 and python3 and then both dependencies added, and then full parallel builds for each side of the chain setup. We're looking at doing that in future versions of ROS but in general the overhead for supporting two dual pipelines significantly increases the amount of maintenance required.

@gbanusi

This comment has been minimized.

gbanusi commented Feb 26, 2018

@jacknlliu I had the same problem on Ubuntu 16.04, and like @tfoote said you need to "get rid" of the python2 packages.

When you execute catkin_make with python3 virtual environment you get this at one point:

first

and you see it is python2.7 -> you need Python3 header files, so execute this line:

sudo apt-get install python3-dev.

After that you will have sth. like this:

second

and then everything works like @randoms said.
Btw. thanks @randoms your helped me alot. :)

@tfoote

This comment has been minimized.

Member

tfoote commented Feb 27, 2018

@gbanusi It's great you've got it working. Thanks for the followup so others won't hit the same issue. One side note please copy and paste console output instead of screenshotting. That way it's searchable and copyable again (say if I want to grep or run find to help you debug I don't have to retype the text).

@jacknlliu FYI there's a bigger discussion of targeting python version here: ros-infrastructure/rep#154

@Kaju-Bubanja

This comment has been minimized.

Kaju-Bubanja commented Mar 16, 2018

@tfoote What exactly is meant by "You need to make sure that all elements are built using only python3."? I found this page, but nowhere do I see an option relating python2 or 3. How would one make sure that all elements are built using only python3?

@gbanusi

This comment has been minimized.

gbanusi commented Mar 16, 2018

Just make a virtual environment for python3 and activate it as @randoms said,
git clone all the packages (tf2, geometry.... you can find them on ros github) you need to work with in catkin_ws/src (also the ones that are dependent, you know that by the error message),
catkin_make --force-cmake will compile everything with found python3 in your virt. env.,
then everything should work....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment