Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extension built with a shared python cannot be loaded with a static python #65735

Closed
pitrou opened this issue May 19, 2014 · 35 comments
Closed
Labels
3.8 stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@pitrou
Copy link
Member

pitrou commented May 19, 2014

BPO 21536
Nosy @malemburg, @loewis, @warsaw, @ncoghlan, @pitrou, @vstinner, @merwok, @xdegaye, @embray, @dstufft
PRs
  • bpo-21536: C extensions are no longer linked to libpython #12946
  • bpo-21536: Revert Makefile change on python-config #12971
  • bpo-21536: On Android, C extensions are linked to libpython #12989
  • bpo-21536: Fix What's New in Python 3.8 entry #13242
  • bpo-21536: On Cygwin, C extensions must be linked with libpython #13549
  • bpo-21536: Follow-up to #13549 #13552
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-04-29.14:44:31.749>
    created_at = <Date 2014-05-19.18:50:02.385>
    labels = ['3.8', 'type-feature', 'library']
    title = 'extension built with a shared python cannot be loaded with a static python'
    updated_at = <Date 2020-12-14.09:17:25.183>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2020-12-14.09:17:25.183>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-04-29.14:44:31.749>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2014-05-19.18:50:02.385>
    creator = 'pitrou'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 21536
    keywords = ['patch']
    message_count = 34.0
    messages = ['218806', '218809', '218813', '219181', '219186', '220293', '340814', '340817', '340818', '340819', '340821', '340837', '340852', '340854', '340870', '340914', '340924', '340926', '340927', '340940', '340973', '340974', '341014', '341046', '341085', '342149', '343372', '343386', '343396', '343397', '343410', '343418', '343420', '343664']
    nosy_count = 13.0
    nosy_names = ['lemburg', 'loewis', 'barry', 'ncoghlan', 'pitrou', 'vstinner', 'eric.araujo', 'xdegaye', 'erik.bray', 'piotr.dobrogost', 'dstufft', 'reimar', 'zhtw1234']
    pr_nums = ['12946', '12971', '12989', '13242', '13549', '13552']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue21536'
    versions = ['Python 3.8']

    @pitrou
    Copy link
    Member Author

    pitrou commented May 19, 2014

    When a C extension is built (using distutils) with a shared library Python, it cannot be loaded with an otherwise identical statically linked Python. The other way round works fine. Trivial example using the _ssl module:

    >>> import sys
    >>> sys.path.insert(0, '/home/antoine/cpython/shared/build/lib.linux-x86_64-3.5/')
    >>> import _ssl
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory

    This is probably because of an additional -L flag that is passed when linking a C extension with a shared library Python. I don't think the flag is useful under Linux (or perhaps under any other OS at all), since the relevant symbols are already loaded when the interpreter tries to load the C extension.

    (AFAIK, systems notorious for providing shared library Pythons are RedHat-alike systems, while Debian/Ubuntu provide statically linked Pythons)

    @pitrou pitrou added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels May 19, 2014
    @pitrou
    Copy link
    Member Author

    pitrou commented May 19, 2014

    Actually, it's not a -L flag but a -l flag. Removing the "-lpython3.5m" flag from the linker line works fine under Linux, and allows the resulting extension to be loaded with both a shared libary Python and a statically-linked Python.

    @loewis
    Copy link
    Mannequin

    loewis mannequin commented May 19, 2014

    I think you are right. It would IMO be useful to research a few comparable systems. E.g. Apache modules don't link a shared library, but still refer to apr_ functions as undefined symbols - but then, there isn't an APR shared library in the first place (at least not on Debian - how about Redhat?)

    PHP might be close to our case: Debian includes a libphp5.so (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so link against it, and all the PHP modules (in /usr/lib/php5/20100525+lfs/) don't link with the shared library - on Debian. I wonder how it is on systems that actually use the PHP shared library.

    @pitrou
    Copy link
    Member Author

    pitrou commented May 26, 2014

    PHP might be close to our case: Debian includes a libphp5.so
    (in /usr/lib/php5), yet neither /usr/bin/php5 nor the Apache libphp5.so
    link against it, and all the PHP modules (in
    /usr/lib/php5/20100525+lfs/) don't link with the shared library -
    on Debian. I wonder how it is on systems that actually use the PHP
    shared library.

    Regardless of libphp5.so, the modules in /usr/lib/php5/20100525 don't link against /usr/lib/apache2/modules/libphp5.so, though the latter is obviously able to load those modules.

    @pitrou
    Copy link
    Member Author

    pitrou commented May 26, 2014

    Hmm, apparently the -l flag was added in bpo-832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).

    (also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library)

    @pitrou
    Copy link
    Member Author

    pitrou commented Jun 11, 2014

    Martin, what do you think about the aforementioned use case?

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    See also bpo-34814.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    Antoine:

    Hmm, apparently the -l flag was added in bpo-832799, for a rather complicated case where the interpreter is linked with a library dlopened by an embedding application (I suppose for some kind of plugin system). The OP there also mentions RTLD_GLOBAL as a workaround (or perhaps the right way of achieving the desired effect).
    (also, the OP didn't mention why he used a shared library build, instead of linking Python statically with the dlopened library)

    bpo-34814 is linked to this use case: https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). Problem: some C extensions of the standard library cannot be loaded in this case, like _struct.

    On Fedora and RHEL, some C extensions like _struct are built by the "*shared*" section of Modules/Setup. In this case, these C extensions are not explicitly linked to libpython.

    IHMO it's a bad usage of dlopen(): libpython must always be loaded with RTLD_GLOBAL.

    bpo-832799 has been fixed by the following commit which modify distutils to link C extensions to libpython:

    commit 10acfd0
    Author: Martin v. Löwis <martin@v.loewis.de>
    Date: Mon Apr 10 12:39:36 2006 +0000

    Patch bpo-1429775: Link Python modules to libpython on linux if
    --enable-shared. Fixes bpo-832799.
    

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    bpo-34814 is linked to this use case: https://bugzilla.redhat.com/show_bug.cgi?id=1585201 is an example of Python embedded in C using dlopen("libpython2.7.so.1.0", RTLD_LOCAL | RTLD_NOW). Problem: some C extensions of the standard library cannot be loaded in this case, like _struct.

    bpo-1429775 is another example of RTLD_LOCAL usage, see:
    https://mail.python.org/pipermail/python-dev/2006-February/060533.html

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    I wrote PR 12946: "On Unix, C extensions are no longer linked to libpython".

    Using PR 12946, the use case described in the initial message (msg218806) now works as expected. I can load a C extension built by a shared library Python with a statically linked Python:

    $ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38notshared/bin/python3.8
    Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:10:57) 
    >>> import sys; sys.path.insert(0, '/opt/py38shared/lib/python3.8/lib-dynload')
    >>> import _ssl
    >>> _ssl
    <module '_ssl' from '/opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'>

    /opt/py38notshared/bin/python3.8 is statically linked, whereas /opt/py38shared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so comes from a shared library Python.

    Install shared libray Python into /opt/py38shared:

    git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix /opt/py38shared && make && make install

    Install statically linked Python into /opt/py38notshared:

    git clean -fdx; ./configure CFLAGS="-O0" --prefix /opt/py38notshared && make && make install

    --

    As Antoine said, the opposite already works. Just in case, I also tested and I confirm that it still works:

    $ LD_LIBRARY_PATH=/opt/py38shared/lib /opt/py38shared/bin/python3.8
    Python 3.8.0a3+ (heads/master:be0099719c, Apr 25 2019, 02:09:02) 
    >>> import sys; sys.path.insert(0, '/opt/py38notshared/lib/python3.8/lib-dynload')
    >>> import _ssl
    >>> _ssl
    <module '_ssl' from '/opt/py38notshared/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so'>

    _ssl comes from statically linked Python (/opt/py38notshared) and is loaded in shared library Python (/opt/py38shared).

    @vstinner vstinner removed the 3.7 label Apr 25, 2019
    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    With an additonal change on SOABI (I will open a separated issue for that), PR 12946 allows to load lxml built in release mode in a Python built in debug mode! That's *very* useful for debugging: see my gdb example below.

    ---

    I just modified the ABI of debug build so release and debug build now have the same ABI: bpo-36465.

    I wrote a patch to use the same sys.implementation.cache_tag (SOABI) in release and debug mode:

    diff --git a/configure b/configure
    index b02d17c053..38eb7f1bd6 100755
    --- a/configure
    +++ b/configure
    @@ -6325,7 +6325,6 @@ $as_echo "#define Py_DEBUG 1" >>confdefs.h
       { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5
     $as_echo "yes" >&6; };
       Py_DEBUG='true'
    -  ABIFLAGS="${ABIFLAGS}d"
     else { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
     $as_echo "no" >&6; }; Py_DEBUG='false'
     fi
    diff --git a/configure.ac b/configure.ac
    index 65d3f8e691..1b2cd3076c 100644
    --- a/configure.ac
    +++ b/configure.ac
    @@ -1223,7 +1223,6 @@ then
       [Define if you want to build an interpreter with many run-time checks.])
       AC_MSG_RESULT(yes);
       Py_DEBUG='true'
    -  ABIFLAGS="${ABIFLAGS}d"
     else AC_MSG_RESULT(no); Py_DEBUG='false'
     fi],
     [AC_MSG_RESULT(no)])

    (That's a temporary patch, I will design a better solution later.)

    Using this patch + PR 12946, it becomes possible to load a C extension compiled in release mode in a debug Python!

    ---

    Example building lxml in release mode and then load it in debug mode.

    Install Python in *release* mode into /opt/py38release (shared libpython):

    git clean -fdx; ./configure --enable-shared --prefix /opt/py38release && make && make install

    Install Python in *debug* mode into /opt/py38debug (shared libpython):

    git clean -fdx; ./configure CFLAGS="-O0" --enable-shared --prefix /opt/py38debug --with-pydebug && make && make install

    Build lxml in release mode:

    LD_LIBRARY_PATH=/opt/py38release/lib/ /opt/py38release/bin/python3.8 -m pip install lxml

    By default, the debug Python doesn't have lxml:

    $ LD_LIBRARY_PATH=/opt/py38debug/lib/ /opt/py38debug/bin/python3.8  -c 'import lxml'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'lxml'

    Give access to C extensions compiled in release mode to the debug Python:

    $ LD_LIBRARY_PATH=/opt/py38debug/lib/ PYTHONPATH=/opt/py38release/lib/python3.8/site-packages /opt/py38debug/bin/python3.8 
    Python 3.8.0a3+ (heads/omit_libpython-dirty:8a03782387, Apr 25 2019, 02:52:01) 
    >>> import lxml
    >>> import lxml.etree
    >>> lxml.etree
    <module 'lxml.etree' from '/opt/py38release/lib/python3.8/site-packages/lxml/etree.cpython-38-x86_64-linux-gnu.so'>
    >>> import sys
    >>> sys.gettotalrefcount()
    108304

    It works!

    Explanation:

    • This is a debug build: sys.gettotalrefcount() is present (and works)
    • lxml has been compiled in release mode
    • etree.cpython-38-x86_64-linux-gnu.so is not linked to libpython and so doesn't rely on the *release* /opt/py38release/lib/libpython3.8.so
    • this lxml can be loaded in a Python compiled in debug mode!

    ---

    Now let's have a look at the gdb experience of release vs debug Python build.

    I put a breakpoint on lxml.etree.iterparse('example.xml'): the C function is called __pyx_tp_new_4lxml_5etree_iterparse.

    Using the release build, gdb fails to read many C local variables:

    $ LD_LIBRARY_PATH=/opt/py38release/lib/ gdb -args /opt/py38release/bin/python3.8 parse.py 

    (gdb) source /home/vstinner/prog/python/master/python-gdb.py
    (gdb) b __pyx_tp_new_4lxml_5etree_iterparse
    Function "__pyx_tp_new_4lxml_5etree_iterparse" not defined.
    Make breakpoint pending on future shared library load? (y or [n]) y
    (gdb) run

    Breakpoint 1, __pyx_tp_new_4lxml_5etree_iterparse (t=0x7fffea724900 <__pyx_type_4lxml_5etree_iterparse>, a=('example.xml',), k=0x0) at src/lxml/etree.c:218930
    218930 src/lxml/etree.c: No such file or directory.

    (gdb) py-bt
    Traceback (most recent call first):
      File "parse.py", line 4, in func
        context = etree.iterparse('example.xml')
      File "parse.py", line 12, in <module>
        func("arg")

    (gdb) frame 4
    #4 _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3268
    3268 res = call_function(&sp, oparg, NULL);
    (gdb) p f
    $1 = <optimized out>
    (gdb) p co
    $2 = <optimized out>
    (gdb) p sp
    $3 = <optimized out>
    (gdb) p oparg
    $4 = <optimized out>

    The basic function "py-bt" works as expected, but inspecting Python internals doesn't work: most local C variables are "optimized out" :-(

    New attempt using a debug build:

    $ LD_LIBRARY_PATH=/opt/py38debug/lib/ PYTHONPATH=/opt/py38release/lib/python3.8/site-packages gdb -args /opt/py38debug/bin/python3.8 parse.py 

    ... same commands to load python-gdb.py and put a breakpoint ...

    Breakpoint 1, __pyx_tp_new_4lxml_5etree_iterparse (t=0x7fffea606900 <__pyx_type_4lxml_5etree_iterparse>, a=('example.xml',), k=0x0) at src/lxml/etree.c:218930
    218930 src/lxml/etree.c: No such file or directory.

    (gdb) py-bt
    Traceback (most recent call first):
      File "parse.py", line 4, in func
        context = etree.iterparse('example.xml')
      File "parse.py", line 12, in <module>
        func("arg")

    (gdb) frame 4
    #4 0x00007ffff7d6e4f8 in _PyEval_EvalFrameDefault (f=Frame 0x458790, for file parse.py, line 4, in func (arg='arg'), throwflag=0) at Python/ceval.c:3268
    3268 res = call_function(&sp, oparg, NULL);
    (gdb) p f
    $1 = Frame 0x458790, for file parse.py, line 4, in func (arg='arg')
    (gdb) p co
    $2 = (PyCodeObject *) 0x7fffea682388
    (gdb) p co->co_consts
    $4 = (None, 'example.xml', 'None', ' => ')
    (gdb) p sp
    $5 = (PyObject **) 0x458938
    (gdb) p sp[0]
    $6 = <module at remote 0x7fffea63a2f0>
    (gdb) p sp[-1]
    $7 = 'example.xml'
    (gdb) p oparg
    $8 = 1

    The debugging experience is *much* better: it's possible to inspect Python internals!

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    If an application is linked to libpython to embed Python, the application build requires flags to link to libpython. Currently, there are:

    $ python3-config --libs
    -lpython3.7m -lcrypt -lpthread -ldl  -lutil -lm 
    
    $ pkg-config --libs python-3.7 
    -lpython3.7m 

    My PR doens't change Misc/python.pc.

    Discussion on python-dev:

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    New changeset 8c3ecc6 by Victor Stinner in branch 'master':
    bpo-21536: C extensions are no longer linked to libpython (GH-12946)
    8c3ecc6

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    If an application is linked to libpython to embed Python, the application build requires flags to link to libpython. (...)

    I created bpo-36721 "Add pkg-config python-3.8-embed" to discuss this issue.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 25, 2019

    Since this change is causing subtle issues like bpo-36721 "Add pkg-config python-3.8-embed", I don't think that it would be a good idea to backport this change to Python 2.7 or 3.7. I close the issue.

    See also bpo-36722 "In debug build, load also C extensions compiled in release mode or compiled using the stable ABI".

    @xdegaye
    Copy link
    Mannequin

    xdegaye mannequin commented Apr 26, 2019

    Please revert the change in Makefile.pre.in made by changeset 8c3ecc6 as it breaks builds made out of the source tree (OST). The error message is:

    make: *** No rule to make target '/path_to_build_dir/Misc/python-config.sh', needed by 'python-config'.  Stop.
    

    When the source tree has been already built at least once and 'make clean' is run in the source tree (as required for building OST), the OST build does not fail but incorrectly uses the stale python-config.sh from the source tree as 'make clean' does not remove Misc/python-config.sh.

    @vstinner
    Copy link
    Member

    vstinner commented Apr 26, 2019

    Please revert the change in Makefile.pre.in made by changeset 8c3ecc6 as it breaks builds made out of the source tree (OST). The error message is:

    make: *** No rule to make target '/path_to_build_dir/Misc/python-config.sh', needed by 'python-config'.  Stop.
    

    (...)

    Oh. I tried to "fix" the Makefile but it seems like I misunderstood how Misc/python-config.sh is handled. This file is generated from Misc/python-config.sh.in.

    I wrote 12971 to revert the change but also add a comment to explain the "magic" behind this file.

    @vstinner vstinner reopened this Apr 26, 2019
    @vstinner
    Copy link
    Member

    vstinner commented Apr 26, 2019

    New changeset 01f073f by Victor Stinner in branch 'master':
    bpo-21536: Revert Makefile change on python-config (GH-12971)
    01f073f

    @vstinner
    Copy link
    Member

    vstinner commented Apr 26, 2019

    I tested manually and I confirm that my latest change fix the compilation out of the source tree. I close again the issue. Sorry for the regression, it's now fixed.

    @xdegaye
    Copy link
    Mannequin

    xdegaye mannequin commented Apr 26, 2019

    Thanks for fixing the regression Victor.

    Here is another potential problem.

    On Android API 24 built with NDK r19, symbol resolution fails in the _socket shared library after changeset 8c3ecc6 when python is built with '--enable-shared':

      generic_x86_64:/data/local/tmp/python $ python -c "import _socket"
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ImportError: dlopen failed: cannot locate symbol "PyByteArray_Type" referenced by "/data/local/tmp/python/lib/python3.8/lib-dynload/_socket.cpython-38d.so"...

    This does not happen when the build is configured without '--enable-shared'.

    An NDK issue [1] reports the same problem and dimitry in this issue explains that on Android >= 23 the shared library must be linked against the shared library where the symbol is defined. Dimitry says that the Android loader was fixed in M (i.e. API 23) and it must refer to the changes listed in the "RTLD_LOCAL (Available in API level >= 23)" section of "Android changes for NDK developers" [2].

    [1] android/ndk#201
    [2] https://android.googlesource.com/platform/bionic/+/android-n-mr2-preview-1/android-changes-for-ndk-developers.md

    @vstinner
    Copy link
    Member

    vstinner commented Apr 27, 2019

    I reopen the issue for the Android case.

    I am fine with having a different linking policy per platform. For example, Windows has a different policy than Linux. We can continue to link C extensions to libpython on Android. The ability to load release C extension in debug Python is a feature more for debugging. I am not sure that Android is the best platform for debugging anyway. The debug can be done on a desktop Linux for example, no?

    @vstinner vstinner reopened this Apr 27, 2019
    @vstinner
    Copy link
    Member

    vstinner commented Apr 27, 2019

    Xavier: would you be interested to work on a fix for Android?

    @xdegaye
    Copy link
    Mannequin

    xdegaye mannequin commented Apr 27, 2019

    PR 12989 fixes the Android issue and has been checked on the Android emulator at API 24.

    Python is cross-compiled with '--enable-shared' and the python-config scripts give the expected result:

    generic_x86_64:/data/local/tmp/python/bin $ python -c "from sysconfig import get_config_var; print(get_config_var('Py_ENABLE_SHARED'))"
    1
    generic_x86_64:/data/local/tmp/python/bin $ sh python-config --libs
    -lpython3.8d -ldl -lm -lm
    generic_x86_64:/data/local/tmp/python/bin $ python python-config.py --libs
    -lpython3.8d -ldl -lm -lm

    @vstinner
    Copy link
    Member

    vstinner commented Apr 29, 2019

    New changeset 254b309 by Victor Stinner (xdegaye) in branch 'master':
    bpo-21536: On Android, C extensions are linked to libpython (GH-12989)
    254b309

    @reimar
    Copy link
    Mannequin

    reimar mannequin commented Apr 29, 2019

    As explained in bpo-34814, this change not only breaks RTLD_LOCAL of libpython, but it breaks it in fact system-wide.
    It seems a bit much for Python to enforce a system-wide policy to not use RTLD_LOCAL just because of its own technical challenges, and thus I'd ask to reconsider the original change.

    @vstinner
    Copy link
    Member

    vstinner commented May 11, 2019

    New changeset 4ebcd7e by Victor Stinner in branch 'master':
    bpo-21536: Update What's New in Python 3.8 entry (GH-13242)
    4ebcd7e

    @embray
    Copy link
    Contributor

    embray commented May 24, 2019

    I vaguely recall seeing some discussion about this on python-dev or elsewhere and wish I had chimed in sooner, as I didn't realize action was going to be taken on this so soon.

    This completely breaks building extension modules on Windows-based platforms like Cygwin and MinGW, where it is necessary when building even shared libraries for all global symbols to resolvable at link time.

    I'm not actually sure this use case can even be achieved on these platforms. I tried several hacks but nothing works. I'll open a follow-up issue...

    @vstinner
    Copy link
    Member

    vstinner commented May 24, 2019

    This completely breaks building extension modules on Windows-based platforms like Cygwin and MinGW, where it is necessary when building even shared libraries for all global symbols to resolvable at link time.

    C extensions are always linked to libpython on Android. I'm open to do something similar for Cygwin and/or MinGW if you can come up with a PR :-) See LIBPYTHON variable in configure.ac.

    You can use Xavier's change as an example: commit 254b309, PR 12989.

    @vstinner
    Copy link
    Member

    vstinner commented May 24, 2019

    New changeset c994c8f by Victor Stinner (E. M. Bray) in branch 'master':
    bpo-21536: On Cygwin, C extensions must be linked with libpython (GH-13549)
    c994c8f

    @vstinner
    Copy link
    Member

    vstinner commented May 24, 2019

    I merged E. M. Bray's PR to get in Python 3.8 beta 1. If anything goes wrong, we can fix it later ;-)

    @vstinner
    Copy link
    Member

    vstinner commented May 24, 2019

    New changeset b1fc417 by Victor Stinner (E. M. Bray) in branch 'master':
    bpo-21536: Fix configure.ac for LIBPYTHON on Android/Cygwin (GH-13552)
    b1fc417

    @xdegaye
    Copy link
    Mannequin

    xdegaye mannequin commented May 24, 2019

    FWIW I checked the Android build after those last changes.

    @vstinner
    Copy link
    Member

    vstinner commented May 24, 2019

    FWIW I checked the Android build after those last changes.

    Thank you! I was overconfident when I merged the Cygwin which looked good to me, whereas it broke Android :-( Hopefully it was quickly fixed. I hope that everything will be alright for the the beta1 to start from a good milestone!

    @embray
    Copy link
    Contributor

    embray commented May 27, 2019

    Thanks everyone. And FWIW I agree the original change is positive overall, if a bit presumptuous about different linkers' behaviors :)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @kpajko79
    Copy link

    kpajko79 commented Sep 2, 2022

    Just ran into this issue while cross-compiling python 3.9.2 for Android API 32 and managed to get the shared extensions working. Made only a few small modifications to the Makefile.

    First added -Wl,-Bsymbolic to all shared libraries in the Android build as the Debian package has a patch to add this to the Linux build (actually -Wl,-Bsymbolic-functions) and i seemed like a good idea to copy that. Mainly because some time ago it was a de-facto standard to add this flag to every native library build [0]. Probably this is unrelated and not required [1].

    Then found an interesting section in the man page of ld:

    global
                   This option is only meaningful when building a shared
                   object.  It makes the symbols defined by this shared
                   object available for symbol resolution of subsequently
                   loaded libraries.
    

    So rebuilt libpython3.9.so with -Wl,-z,global
    And voila, it rocks! Virtually this marks the shared library to be treated as if it had been loaded with RTLD_GLOBAL, moving all its symbols to the global namespace.
    The parameter just turns on a flag in the ELF header, so it could be possible to try its effect with an ELF editor much faster than recompiling.

     0x000000000000001e (FLAGS)              SYMBOLIC BIND_NOW
     0x000000006ffffffb (FLAGS_1)            Flags: NOW GLOBAL
    

    [0] https://gcc.gcc.gnu.narkive.com/ceGm0WCb/android-the-reason-why-bsymbolic-is-turned-on-by-default
    [1] https://www.technovelty.org/c/what-exactly-does-bsymblic-do.html

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants