Python: Deterministic position of libarrow in manylinux1 wheels

[asfimport]

To link other Python wheels to the pyarrow, the position and name of all shared libraries should be deterministic. Currently using libarrow in other Python packages is only possible using the conda packaging stack, via pip installable paths.

Reporter: Uwe Korn / @xhochy
Assignee: Uwe Korn / @xhochy

_{Note: This issue was originally created as ARROW-566. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Well, the shared libraries need to end up effectively in $LD_LIBRARY_PATH or loaded dynamically – conda works as a tool for building a LD_LIBRARY_PATH for the libarrow shared libraries. One option you can use is to load the libpyarrow.so using ctypes based on the absolute module path of pyarrow (this is similar to how NumPy's C-level interoperability works). I presume you're statically-linking the dependencies in pyarrow?

[asfimport]

Uwe Korn / @xhochy:
libarrow* and libparquet* are dynamically linked currently in the wheels. The correct RPATH is set via auditwheel but also a hash is appended at the end of the libname so that different wheels can have different versions of the same library. As pyarrow is the main authority for libarrow in a pip-based setup, there should be no need for the hash-based suffix. The main things to tackle in this issue are:

• Set the correct RPATHs in libpyarrow and libparquet_arrow
• Bundle the above shared libraries in pyarrow.

My main intention is to build turbodbc wheels based upon pyarrow wheels but this should also be needed for any python package consuming both the C++ and Python API of Arrow.

[asfimport]

Uwe Korn / @xhochy:
PR: #360

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 360
#360