ImportError: _ssl.so: undefined symbol: _PyModule_Add

[0x2b3bfa0]

Bug report

Environment

$ uname
Linux
$ python --version{,}
Python 3.11.4 (tags/v3.11.4:d2340ef257, Aug 26 2023, 22:42:28) [GCC 12.2.0]

Description

Importing the ssl shared library from a Python 3.11.5 build from a Python 3.11.4 binary produces an ImportError due to an undefined symbol. Isn't CPython's ABI supposed to be fully compatible between patch releases?

CPython’s Application Binary Interface (ABI) is forward- and backwards-compatible across a minor release.

Example

FROM debian

RUN apt update\
 && apt install --yes build-essential git libssl-dev zlib1g-dev

RUN git clone --branch v3.11.4 https://github.com/python/cpython 3.11.4\
 && cd 3.11.4 && ./configure && make

RUN git clone --branch v3.11.5 https://github.com/python/cpython 3.11.5\
 && cd 3.11.5 && ./configure && make

RUN mv 3.11.5/build/lib.linux-x86_64-3.11/_ssl.cpython-311-x86_64-linux-gnu.so\
       3.11.4/build/lib.linux-x86_64-3.11/_ssl.cpython-311-x86_64-linux-gnu.so

RUN 3.11.4/python -m ssl

Error

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "//3.11.2/Lib/ssl.py", line 100, in <module>
    import _ssl             # if we can't import it, let the error propagate
    ^^^^^^^^^^^
ImportError: //3.11.2/build/lib.linux-x86_64-3.11/_ssl.cpython-311-x86_64-linux-gnu.so: undefined symbol: _PyModule_Add

Observations

It looks like fced79f is the breaking commit, as per git bisect results.

Linked PRs

• gh-108525: Clarify stable ABI documentation to exclude internal C libraries #108582

[hansegucker]

I also observed the issue on my system.

[CorvinM]

Also able to reproduce between 3.11 branch and 3.11.4, looks like #106863 broke the ABI

[sunmy2019]

@serhiy-storchaka as the commit author

[serhiy-storchaka]

I don't think this warranty extends to the private API.

Names prefixed by an underscore, such as _Py_InternalState, are private API that can change without notice even in patch releases.

If you only use the public C API in your extension, you can expect forward- and backwards-compatibinility.

[Templarrr]

I see the same error locally in poetry after my Ubuntu upgraded 3.11.? -> 3.11.5. Any advice how this can be mitigated/fixed? It doesn't look like I can downgrade local python back to 3.11.4 - deadsnakes ppa don't list 3.11.4 anymore

[LostInDarkMath]

@Templarrr I had the same issue and fixed it by updating poetry to the latest version.

[erlend-aasland]

cc. RM @pablogsal

[Templarrr]

@LostInDarkMath thank you! Seems to be the solution

[encukou]

The reproducer looks contrived. In what situation would pieces of CPython get mixed up like this? Is Ubuntu's _ssl module not built together with the interpreter?

[CorvinM]

The reproducer looks contrived. In what situation would pieces of CPython get mixed up like this? Is Ubuntu's _ssl module not built together with the interpreter?

A faulty install or packaging script could easily cause this (but wouldn't be noticed in most cases, since they are usually ABI compatible) because the file paths are named identically (I.e., specific to 3.11, not 3.11.5).

[CorvinM]

In this case it looks like a problem with poetry packaging (or its pythonpath?)

[0x2b3bfa0]

The original issue was also related to Python + Poetry, by the way.

[0x2b3bfa0]

A faulty install or packaging script could easily cause this [...] because the file paths are named identically (i.e., specific to 3.11, not 3.11.5)

This. ☝🏼 💯

[zvolsky]

@LostInDarkMath: I had the same issue and fixed it by updating poetry to the latest version.

My poetry --version was 1.6.1.
poetry self update exactly as other poetry commands has failed with this issue.
I decided to "update" via curl -sSL https://install.python-poetry.org | python3 -, i.e. reinstall,
the poetry --version is now again 1.6.1.
And Poetry works again.
(Debian Testing)

[dusty-phillips]

I encountered this issue in a python3.11-slim Dockerfile even after updating poetry.

In my case, the issue was that the Dockerfile in question was installing the python3-dev apt package, which was providing 3.11.2 headers. I was able to remove this dependency without any repercussions.

Was devilishly hard to track down so thought I'd share in case it helps someone else.

[cove9988]

Yes, the old poetry 1.6.1 is not working well with python 3.11
if you curl -sSL https://install.python-poetry.org | python3 - upgrade to 1.7.1, It works well.

[vstinner]

Importing the ssl shared library from a Python 3.11.5 build from a Python 3.11.4 binary produces an ImportError due to an undefined symbol.

Usually, a Python binary loads its "own" Python stdlib. I don't think that we want to support using a newer/older Python with newer/older stdlib. You should use the exact same version for the Python binary and its stdlib.

What can be done is to enhance error reporting. For example, but storing the version in stdlib C extensions and raise an error if there is version mismatch.

Today, you get an ImportError because of a missing symbol. But tomorrow, you might get a crash because of subtle different behavior :-(

[vstinner]

I close the issue. I don't think that we want to support this use case.