ctypes.util.find_library should examine binary's RPATH on Solaris

[automatthias]

BPO	19317
Nosy	@jcea, @vadmium
Files	find_library_looks_into_rpath.patch: Example implementation

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-10-20.15:31:38.805>
labels = ['ctypes', 'type-feature']
title = "ctypes.util.find_library should examine binary's RPATH on Solaris"
updated_at = <Date 2016-04-28.06:43:36.911>
user = 'https://bugs.python.org/automatthias'

bugs.python.org fields:

activity = <Date 2016-04-28.06:43:36.911>
actor = 'martin.panter'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2013-10-20.15:31:38.805>
creator = 'automatthias'
dependencies = []
files = ['32259']
hgrepos = []
issue_num = 19317
keywords = ['needs review']
message_count = 12.0
messages = ['200593', '200863', '200864', '200865', '200889', '256730', '264147', '264349', '264355', '264412', '264413', '264419']
nosy_count = 4.0
nosy_names = ['jcea', 'automatthias', 'martin.panter', 'risto3']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19317'
versions = ['Python 2.6', 'Python 2.7', 'Python 3.3', 'Python 3.4']

[automatthias]

On Solaris, when you want to link shared libraries from custom directories, you most often don't modify the system search path, but instead set RPATH in your binaries. For example, OpenCSW packages Python into /opt/csw, and sets Python executable's RPATH to /opt/csw/lib. Therefore, dynamically opening shared libraries will by default look into /opt/csw/lib first, and find_library should do the same. I wrote a sample implementation.

[jcea]

Patch looks good. Any other reviewer?

Maciej, do you know about any generally available library in Solaris with SONAME and RPATH?. A test would be nice.

Would this patch solve the "locate the C library" using ctypes, in Solaris? :-).

[jcea]

"/usr/bin/chkpass.so" has both SONAME and RPATH.

Would you mind, anyway, to elaborate a bit your use case, including some example?. Thanks!.

[jcea]

I mean "/usr/lib/chkpass.so".

[automatthias]

The specific case that breaks for me is this:

OpeCSW Python package includes:

/opt/csw/bin/python2.6 (also 2.7, 3.3, etc)
/opt/csw/lib/libpython2.6.so.1.0
(other files)

On the operating system there is only:
/usr/lib/libpython2.4.so.1.0

Let's say there's libmagic installed in /opt/csw/lib, and there's no libmagic in /usr/lib. Let's suppose you run this:

ctypes.cdll.LoadLibrary('libmagic.so.1')

The OpenCSW Python will successfully load libmagic.so.1. But without the patch, find_library will not find libmagic.so.1, it will fail to translate 'magic' to 'libmagic.so.1', even though the libmagic.so symlink is present in /opt/csw/lib.

For the patch, the RPATH of the library itself doesn't matter, the important one is the RPATH of the Python executable itself, which influences what _ctypes.dlopen() does when looking for a library.

I could write a test by providing providing a sample /usr/ccs/bin/dump output and mocking out some libraries. Would that be good?

[risto3]

I tried this patch out on pkgsrc, it does seem reasonable and appropriate. So +1 from me.

It does only look for libraries the actual $PREFIX directory used by packaging systems such as pkgsrc and csw. (typically /usr/local, /opt/local or /opt/csw in the case of csw)

So for completeness, perhaps consideration should be made some time for cases such as when environment variables like LD_LIBRARY_PATH*, LD_CONFIG* et cetera are used... BTW, LD_LIBRARY_PATH is also used on other ELF platforms.

[vadmium]

find_library() is documented as emulating the build-time linker, and I suspect the RPATH of the Python executable is not relevant at build time. See also bpo-9998 and bpo-18502.

[risto3]

There *is* a feature with linking called $ORIGIN.

[risto3]

oups... I meant to add the comment about $ORIGIN (not really useful here) but also the fact that the binary python is built with the dependencies found via the library path (-L, for example) and the eventual run-paths (-R or -runpath) when not in the system paths.

As indicated by automatthias@, package managers such as pkgsrc, CSW, *ports and the like have their $PREFIX off of the system main path.. usually in /usr/pkg, /usr/local, /opt/local or something like that
(for unices and linux sort of systems, anyway).

My pkgsrc python2.7 binary has the following runpath:

richard@omnis:/home/richard$ dump -Lpv /opt/local/bin/python2.7 |grep RPATH
[12]	RPATH           /opt/pbulk32/gcc49/i486-sun-solaris2.11/lib/.:/opt/pbulk32/gcc49/lib/.:/opt/local/lib

What is being requested greatly simplifies things, and can probably be generalised to other ELF based systems too.

BTW I came across the issue as well with py-magic.

[risto3]

An example:

richard@omnis:/home/richard$ python2.7
Python 2.7.11 (default, Apr 27 2016, 04:35:25) 
[GCC 4.9.3] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import *
>>> from ctypes.util import find_library
>>> find_library('magic')
>>> cdll.LoadLibrary('libmagic.so')
<CDLL 'libmagic.so', handle fe240840 at fe7879ac>

Finally, as you can see above, LoadLibrary() works fine to load '/opt/local/libmagic.so' because of the runpath in the python binary, but find_library() does not because the runpath is ignored.

This should probably be considered as 'unexpected' behaviour.

[risto3]

[fingers not yet warmed up] that is '/opt/local/lib/libmagic.so'

[vadmium]

I am realizing that many people like to use find_library() as a way of converting a portable name like “magic” (from building with -lmagic) into a platform-specific name (libmagic.so.1, libmagic.dylib, magic.dll, etc), and then pass this to LoadLibrary() or equivalent. So searching Python’s runpath would probably be valid for that use case.

IMO the extra searching is inefficient, and not robust if there is more than one version of the library. Personally I would be more interested in adding an alternative function, maybe make_library_name("magic", "1") -> "libmagic.so.1", or cdll.LoadLibraryByPortableName("magic", "1").