extension built with a shared python cannot be loaded with a static python

[pitrou]

BPO	21536
Nosy	@malemburg, @loewis, @warsaw, @ncoghlan, @pitrou, @vstinner, @merwok, @xdegaye, @embray, @dstufft
PRs	bpo-21536: C extensions are no longer linked to libpython #12946 bpo-21536: Revert Makefile change on python-config #12971 bpo-21536: On Android, C extensions are linked to libpython #12989 bpo-21536: Fix What's New in Python 3.8 entry #13242 bpo-21536: On Cygwin, C extensions must be linked with libpython #13549 bpo-21536: Follow-up to #13549 #13552

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-29.14:44:31.749>
created_at = <Date 2014-05-19.18:50:02.385>
labels = ['3.8', 'type-feature', 'library']
title = 'extension built with a shared python cannot be loaded with a static python'
updated_at = <Date 2020-12-14.09:17:25.183>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2020-12-14.09:17:25.183>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-29.14:44:31.749>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2014-05-19.18:50:02.385>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 21536
keywords = ['patch']
message_count = 34.0
messages = ['218806', '218809', '218813', '219181', '219186', '220293', '340814', '340817', '340818', '340819', '340821', '340837', '340852', '340854', '340870', '340914', '340924', '340926', '340927', '340940', '340973', '340974', '341014', '341046', '341085', '342149', '343372', '343386', '343396', '343397', '343410', '343418', '343420', '343664']
nosy_count = 13.0
nosy_names = ['lemburg', 'loewis', 'barry', 'ncoghlan', 'pitrou', 'vstinner', 'eric.araujo', 'xdegaye', 'erik.bray', 'piotr.dobrogost', 'dstufft', 'reimar', 'zhtw1234']
pr_nums = ['12946', '12971', '12989', '13242', '13549', '13552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21536'
versions = ['Python 3.8']

[pitrou]

When a C extension is built (using distutils) with a shared library Python, it cannot be loaded with an otherwise identical statically linked Python. The other way round works fine. Trivial example using the _ssl module:

>>> import sys
>>> sys.path.insert(0, '/home/antoine/cpython/shared/build/lib.linux-x86_64-3.5/')
>>> import _ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

This is probably because of an additional -L flag that is passed when linking a C extension with a shared library Python. I don't think the flag is useful under Linux (or perhaps under any other OS at all), since the relevant symbols are already loaded when the interpreter tries to load the C extension.

(AFAIK, systems notorious for providing shared library Pythons are RedHat-alike systems, while Debian/Ubuntu provide statically linked Pythons)

[pitrou]

Actually, it's not a -L flag but a -l flag. Removing the "-lpython3.5m" flag from the linker line works fine under Linux, and allows the resulting extension to be loaded with both a shared libary Python and a statically-linked Python.

[loewis]

I think you are right. It would IMO be useful to research a few comparable systems. E.g. Apache modules don't link a shared library, but still refer to apr_ functions as undefined symbols - but then, there isn't an APR shared library in the first place (at least not on Debian - how about Redhat?)

PHP might be close to our case: Debian includes a libphp5.so (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so link against it, and all the PHP modules (in /usr/lib/php5/20100525+lfs/) don't link with the shared library - on Debian. I wonder how it is on systems that actually use the PHP shared library.

[pitrou]

PHP might be close to our case: Debian includes a libphp5.so
(in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so
link against it, and all the PHP modules (in
/usr/lib/php5/20100525+lfs/) don't link with the shared library -
on Debian. I wonder how it is on systems that actually use the PHP
shared library.

Regardless of libphp5.so, the modules in /usr/lib/php5/20100525 don't link against /usr/lib/apache2/modules/libphp5.so, though the latter is obviously able to load those modules.

[pitrou]

Hmm, apparently the -l flag was added in bpo-832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).

(also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library)

[pitrou]

Martin, what do you think about the aforementioned use case?

[vstinner]

See also bpo-34814.

[vstinner]

Antoine:

Hmm, apparently the -l flag was added in bpo-832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).
(also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library)

bpo-34814 is linked to this use case: https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). Problem: some C extensions of the standard library cannot be loaded in this case, like _struct.

On Fedora and RHEL, some C extensions like _struct are built by the "*shared*" section of Modules/Setup. In this case, these C extensions are not explicitly linked to libpython.

IHMO it's a bad usage of dlopen(): libpython must always be loaded with RTLD_GLOBAL.

bpo-832799 has been fixed by the following commit which modify distutils to link C extensions to libpython:

commit 10acfd0
Author: Martin v. Löwis <martin@v.loewis.de>
Date: Mon Apr 10 12:39:36 2006 +0000

Patch bpo-1429775: Link Python modules to libpython on linux if
--enable-shared. Fixes bpo-832799.

[vstinner]

bpo-34814 is linked to this use case: https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). Problem: some C extensions of the standard library cannot be loaded in this case, like _struct.

bpo-1429775 is another example of RTLD_LOCAL usage, see:
https://mail.python.org/pipermail/python-dev/2006-February/060533.html

[vstinner]

I wrote PR 12946: "On Unix, C extensions are no longer linked to libpython".

Using PR 12946, the use case described in the initial message (msg218806) now works as expected. I can load a C extension built by a shared library Python with a statically linked Python:

$ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38notshared/bin/python3.8
Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:10:57) 
>>> import sys; sys.path.insert(0, '/opt/py38shared/lib/python3.8/lib-dynload')
>>> import _ssl
>>> _ssl
<module '_ssl' from '/opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'>

/opt/py38notshared/bin/python3.8 is statically linked, whereas /opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so comes from a shared library Python.

Install shared libray Python into /opt/py38shared:

git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix /opt/py38shared && make && make install

Install statically linked Python into /opt/py38notshared:

git clean -fdx; ./configure CFLAGS="-O0" --prefix /opt/py38notshared && make && make install

--

As Antoine said, the opposite already works. Just in case, I also tested and I confirm that it still works:

$ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38shared/bin/python3.8
Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:09:02) 
>>> import sys; sys.path.insert(0, '/opt/py38notshared/lib/python3.8/lib-dynload')
>>> import _ssl
>>> _ssl
<module '_ssl' from '/opt/py38notshared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'>

_ssl comes from statically linked Python (/opt/py38notshared) and is loaded in shared library Python (/opt/py38shared).