Bindings for a shared library

[mdimura]

If I am wrapping a shared library (e.g. libParent.so), and generate python bindings (e.g. libBindings.so), setup.py will install both libBindings.so (correctly) and libParent.so (incorrectly) to python's .../dist-packages/. .../dist-packages/ directory is not usually in $LD_LIBRARY_PATH, so the loader does not see libParent.so and I get an error like this:

ImportError: libParent.so.0.0.1: cannot open shared object file: No such file or directory

Contents of Bindings/CMakeLists.txtlook like this:

pybind11_add_module(Bindings bindings.cpp)
target_link_libraries(Bindings PRIVATE Parent)

Is there an elegant way to solve this?

[dead]

Same problem.... Have you found a solution?

[mdimura]

Not really a solution, but there are two possible workarounds: install libParent.so separately as a dependency, or link libParent.so statically into libBindings.so.

[dead]

Lol, I think I just found out a good solution, just compile with -DCMAKE_INSTALL_RPATH=$ORIGIN/.

Nop, this link to the build folder .so

[eirrgang]

For a library that is built strictly to support the Python bindings, I decided the most appropriate solution was to build it as a static library and link it into the Python module shared object. E.g.

add_library(supporting_library STATIC ...)
set_target_properties(supporting_library PROPERTIES POSITION_INDEPENDENT_CODE ON)
...
pybind11_add_module(pymodule MODULE ...)
target_link_libraries(pymodule PRIVATE supporting_library)

If you want to install both the supporting library and the Python module as shared objects, the RPATH is harder to manage across different OSes, but what works for me is something like the following:

set_target_properties(pymodule PROPERTIES SKIP_BUILD_RPATH FALSE)

\# If building with setuptools, CMake will not be performing the install
set_target_properties(pymodule PROPERTIES BUILD_WITH_INSTALL_RPATH TRUE)

target_link_libraries(pymodule PRIVATE supporting_library)

if(UNIX AND NOT APPLE)
    set_target_properties(pymodule PROPERTIES INSTALL_RPATH "$ORIGIN/relative/path/to/supporting/library_dir")
elseif(APPLE)
    set_target_properties(pymodule PROPERTIES INSTALL_RPATH "@loader_path/relative/path/to/supporting/library_dir")
endif()

set_target_properties(pymodule PROPERTIES INSTALL_RPATH_USE_LINK_PATH TRUE)

In this case the installed paths mirror the build paths. If that is not feasible, then you probably want to separate the libraries into separate projects, build and install one, then build the python module against the installed library.

[ax3l]

@mdimura @dead I just posted a solution for dependent, shared libraries in #16. Enjoy!

[pythonguru102]

Help me!!!
I'd like to make binding file alprstream.py for alprstream.so.
But Some errors were happend.
Like follow one:

If call functions in alprstream.so, These errors occur.

Traceback (most recent call last):
File "alprstream_images.py", line 49, in
alpr_stream = AlprStream(VIDEO_BUFFER_SIZE, USE_MOTION_DETECTION)
File "/usr/lib/python2.7/dist-packages/openalpr/alprstream.py", line 62, in init
self._initialize_func = self._alprstreampy_lib.initialize
File "/usr/lib/python2.7/ctypes/init.py", line 375, in getattr
func = self.getitem(name)
File "/usr/lib/python2.7/ctypes/init.py", line 380, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/libalprstream.so.3: undefined symbol: initialize

[leimao]

For a library that is built strictly to support the Python bindings, I decided the most appropriate solution was to build it as a static library and link it into the Python module shared object. E.g.

add_library(supporting_library STATIC ...)
set_target_properties(supporting_library PROPERTIES POSITION_INDEPENDENT_CODE ON)
...
pybind11_add_module(pymodule MODULE ...)
target_link_libraries(pymodule PRIVATE supporting_library)

^^^
This solution works fine. Is the logic behind that Pybind11 only want static library to be linked because Python want the library to be portable?

[ax3l]

No, your python module (which is just a shared library with an entry point) can depend on shared libraries at runtime just like any other shared library. Such dependencies will just complicate your install, since you have to ship & find them, too (e.g. via package managers).

[eirrgang]

@leimao: Like @ax3l says, if you don't have to build a separate shared library, it is easier to avoid it. Not only does it complicate the install, but it also means you have to make sure that RPATH gets set right (or use LD_LIBRARY_PATH), which was the gist of the original issue. But note that a static library can only be linked into a shared library (the Python module) if the static library is compiled with POSITION_INDEPENDENT_CODE, which may not be available in some architectures.

[sarnold]

The PIC thing is indeed required for building/linking a static lib, however, you should not need to muck around with RPATH, LD_LIBRARY_PATH, etc. If you are building python bindings for an external lib, then you should be using find_package as needed in your CMakeLists.txt (if the target package uses cmake then it should install the right config bits for that to work). Depending on the library you may need to use the cmake pkg-config support instead, but either way should work (some cmake-built packages even install a .pc file to use). A nice working example (adapted from here) is: https://github.com/freepn/google-re2 which works across the 3 main github CI runners (with only a little jiggering for WIndows).

[gmolkho]

Seems that using the target shared object that include pybind as-is, at the same folder as the dependent one, with prefix "lib" at its name, makes it to become visible without adding any search path info. anywhere (credit to Evgeniy Foox)