[Unix/Linux] Add a configure option to set the RPATH to `$ORIGIN/../lib`

[koubaa]

Feature or enhancement

Proposal:

Hello!

When building python from source, --enable-shared can be used to build dynamic libraries. If this is done, and --prefix is set to a nonstandard location, users may need to set LD_LIBRARY_PATH so that python3 can find libpython3.

An alternative, setting LD_RUN_PATH at build time can be used to hard-code the location to {--prefix}/lib so that at runtime LD_LIBRARY_PATH does not need to be set. But this makes the python installation no longer portable. I cannot install to --prefix and then move it elsewhere.

RPATH is an option in ELF files to provide relatives paths so that an exe can find associated dynamic libraries. In python's case, on *UNIX platforms, this could be "$ORIGIN/../lib". $ORIGIN in an ELF binary refers to the location of the loaded program or dynamic library which is looking for a dependency, in this case the python3 program.

I solved this by setting this variable in bash before running the build (escaping $ORIGIN is a pain!):
export LDFLAGS="-Wl,-rpath,'\$\$ORIGIN/../lib' -z origin -Wl,--disable-new-dtags"

And now, if I read the ELF I find that the RPATH is correctly configured:

$>readelf -d /path/to/install/prefix/bin/python3 | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib]

The problem is, RPATH is also set for every library that comes with python, which may not be desirable (since $ORIGIN is the parent path of the executable first loaded by libdl, not necessarily python3, and libpython3 might be embedded in another program!):

$> readelf -d /inst2/lib/libpython3.10.so | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../lib]

Figuring out how to do this (or even what to do) wasn't trivial.

My proposal:

• An option in the configure script, like --use-rpath-libpython, to build the python program with an rpath set up to its relative lib folder for Unix/Linux.
• Consider making this option the default when building with --enable-shared

As an aside, I don't build on MacOS and know next to nothing about the platform but if the same thing applies there I can update the title.

Links to related previous discussion:

https://modwsgi.readthedocs.io/en/develop/user-guides/installation-issues.html?highlight=ld_run_path#unable-to-find-python-shared-library
https://stackoverflow.com/questions/37757314/problems-installing-python-3-with-enable-shared
docker-library/python#21
pyenv/pyenv#819
https://developer.squareup.com/blog/building-portable-binaries/

[aitzkora]

If you want do do the same thing without recompiling you can do

patchelf --set-rpath "\$ORIGIN/../lib" /path/to/install/prefix/bin/python3