Building with Python3 links against Boost Python2 library

[daniel-j-h]

On a Ubuntu 16.04 box where I have both Python2 as well as Python3:

$ ls /usr/lib/x86_64-linux-gnu/libboost_python*.so
/usr/lib/x86_64-linux-gnu/libboost_python-py27.so
/usr/lib/x86_64-linux-gnu/libboost_python-py35.so
/usr/lib/x86_64-linux-gnu/libboost_python.so

$ ll /usr/lib/x86_64-linux-gnu/libboost_python.so
lrwxrwxrwx 1 root root 49 Dec 22 14:16 /usr/lib/x86_64-linux-gnu/libboost_python.so -> /usr/lib/x86_64-linux-gnu/libboost_python-py27.so

When I install the osmium Python bindings with Python3 and compile osmium from source

python3 -m pip install osmium

the resulting shared object links against Python2:

ldd /usr/local/lib/python3.6/dist-packages/osmium/_osmium.cpython-36m-x86_64-linux-gnu.so | grep python
        libboost_python-py27.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_python-py27.so.1.58.0 (0x00007f91eeff3000)

which results in undefined symbol errors when trying to use osmium from a Python3 script:

ImportError: /usr/lib/x86_64-linux-gnu/libboost_python-py27.so.1.58.0: undefined symbol: PyClass_Type

Is there a way for pyosmium to check if the build process is meant for Python2 or Python3 and then link against 2.x or 3.x Boost.Python shared libraries, respectively?

---

Workaround is a bit ugly but does the job:

python3 -m pip uninstall osmium
rm -rf ~/.cache/pip/wheels/

rm /usr/lib/x86_64-linux-gnu/libboost_python.so
ln -s /usr/lib/x86_64-linux-gnu/libboost_python-py35.so /usr/lib/x86_64-linux-gnu/libboost_python.so

python3 -m pip install osmium

[lonvia]

setup.py already tries to get the right boost version, see here. It rather looks like you are running into a version incompatibility within python3 itself. Your library seems to be built for python 3.6 (note the 36m in /usr/local/lib/python3.6/dist-packages/osmium/_osmium.cpython-36m-x86_64-linux-gnu.so) but you only have a boost library for 3.5 around.

I don't know of any way to check the version of the boost library to catch this kind of problem early in the installation process. Maybe it is possible to compile a small check program. If you have an idea, it would be a most welcome inclusion to the setup process.

[daniel-j-h]

Ah you are right, Ubuntu 16.04 comes with Python 3.5 by default and we were grabbing Python 3.6 from a ppa. The ppa only ships Python 3.6 but not Boost.Python for Python 3.6.

Hrm this is an unfortunate issue on my side then, sorry for the noise. Feel free to close.

[lonvia]

My favorite 'solution' would be to switch to pybind11. It being header-only and all would solve at least one of the dependency issues.

Anyway, closing here for now. An explicit check for version compatibility in setup.py would be a welcome contribution but it is unlikely going to happen from my side.

[jjmata]

How did you resolve this on your end, @daniel-j-h? Trying to run robosat inside one of the latest tag PyTorch builds from here I run smack into this issue. Thanks for any guidance!

[daniel-j-h]

The initial post above has a workaround at the bottom.

(In the context of robosat what you also can do is remove the extract import and setup in robosat/tools/__main__.py We only need osmium for turning features in osm.pbf files into polygons. And you could do that e.g. in a Docker container with Ubuntu 16.04 and its default Python 3.5)

[jjmata]

Thanks @daniel-j-h ... applied your patch to the quick-n-dirty docker-pytorch I got going here: mapbox/robosat#29 (comment)

Sorry I missed it in the first place.