Python 3.11.0 ssl module configuration error on macOS

[mjsteinbaugh]

Bug report

Hi, I'm having trouble getting the configure script to correctly detect OpenSSL for the ssl module, specifically on macOS with Python 3.11.0. Configuration of the ssl module works on Linux, with the same set of isolated dependencies. For reference, this same install script works correctly for 3.10.8.

See relevant code here:
https://github.com/acidgenomics/koopa/blob/6695aac42553dadac2f75668852d318159edd3b5/lang/shell/bash/include/install/common/shared/python.sh

The configuration looks correct in the generated Makefile.pre file:

# OpenSSL options for setup.py so sysconfig can pick up AC_SUBST() vars.
OPENSSL_INCLUDES=-I/opt/koopa/app/openssl3/3.0.5/include
OPENSSL_LIBS=-lssl -lcrypto
OPENSSL_LDFLAGS=-L/opt/koopa/app/openssl3/3.0.5/lib
OPENSSL_RPATH=/opt/koopa/app/openssl3/3.0.5/lib

I'm seeing these OpenSSL-related issues during configuration:

checking whether OpenSSL provides required ssl module APIs... no
checking whether OpenSSL provides required hashlib module APIs... no
[...]
MODULE__SSL_STATE = "missing"

# >>> import ssl
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/opt/koopa/app/python/3.11.0/libexec/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 100, in <module>
#     import _ssl             # if we can't import it, let the error propagate
#     ^^^^^^^^^^^
# ModuleNotFoundError: No module named '_ssl'

# > WARNING: pip is configured with locations that require TLS/SSL,
# > however the ssl module in Python is not available.
# > "Can't connect to HTTPS URL because the SSL module is not available.

Digging into the configure code, it looks like these lines are relevant to the breaking changes I'm observing:

• https://github.com/python/cpython/blob/3.10/configure#L17844
• https://github.com/python/cpython/blob/3.11/configure#L23194

Your environment

• CPython versions tested on: 3.11.0
• Operating system and architecture: macOS 12.6 (x86)

I can provide additional log files or code if that will help debug the issue further.

Best,
Mike

[tpham3783]

I had the same problem building the code base of python 3.8.6 and also 3.8.10 on Rocky8.

I traced it down and found out that the ssl module got built successfully; however, the problem was that the Python Setup.py script excluded installation of the _ssl module because it did not pass an import test. Part of the issue is that python setup.py script relied on the environment variable OPENSSL_LDFLAGS, which was not defined if openssl was installed in the default system config.

I fixed this issue by applying this patch, please review and give feedback.

--- configure.old	2022-10-25 14:13:13.616179815 -0400
+++ configure	2022-10-25 14:12:11.828979774 -0400
@@ -17203,7 +17203,7 @@
 fi
 
             if test x"$PKG_CONFIG" != x""; then
-                OPENSSL_LDFLAGS=`$PKG_CONFIG openssl --libs-only-L 2>/dev/null`
+                OPENSSL_LDFLAGS=`$PKG_CONFIG openssl --libs-only-L --keep-system-libs 2>/dev/null`
                 if test $? = 0; then
                     OPENSSL_LIBS=`$PKG_CONFIG openssl --libs-only-l 2>/dev/null`
                     OPENSSL_INCLUDES=`$PKG_CONFIG openssl --cflags-only-I 2>/dev/null`

[mjsteinbaugh]

@tpham3783 Interesting thanks, I'll try this out and will post an update

[debohman]

Note that the latest version of pkg-config (0.29.2) does not contain an option --keep-system-libs. What version of pkg-config is being referenced here?

[debohman]

I see:

% pkg-config openssl --libs
-L/usr/local/lib -lssl -lcrypto
% pkg-config openssl --libs --keep-system-libs
Unknown option --keep-system-libs
% pkg-config --version                        
0.29.2

[tpham3783]

Note that the latest version of pkg-config (0.29.2) does not contain an option --keep-system-libs. What version of pkg-config is being referenced here?

I am using pkgconfig version 1.4.2 on Rocky-8.

and you are right, my Ubuntu-20.04 has version 0.29.1; it doesn't have that option.

[debohman]

I am not familiar with Rocky-8. Is there is a modified version of pkg-config?

[tpham3783]

I am not familiar with Rocky-8. Is there is a modified version of pkg-config?

Rocky-8 is a linux distribution similar to RedHat and Centos.
No, it is has an updated version of pkg-config. Are you having the same issue too?

[debohman]

Rocky-8 is a linux distribution similar to RedHat and Centos.
No, it is has an updated version of pkg-config. Are you having the same issue too?

No, I am not having an issue with Python-3.11.0. My concern is that is that we need to be careful if we pass platform specific options to pkg-config.

[tpham3783]

As stated earlier, the root cause of the bug was from setup.py not being able to verify the ssl module even though it was built successfully. You can test to see if you ran into the same bug with the small patch below. Again, I am just sharing how I fixed it.

--- setup.py.old	2022-10-25 17:19:50.929018770 -0400
+++ setup.py	2022-10-25 17:21:01.119521395 -0400
@@ -479,6 +479,7 @@
             return
 
     def check_extension_import(self, ext):
+        return
         # Don't try to import an extension that has failed to compile
         if ext.name in self.failed:
             self.announce(

[debohman]

Okay. I do have openssl 3.0 present in my build of Python-3.11.0.

[mjsteinbaugh]

Passing these variables to configure works nicely:

ac_cv_working_openssl_hashlib=yes
ac_cv_working_openssl_ssl=yes

Python 3.11.0 installed successfully! 🚀