ssl/hashlib: Add configure option to set or auto-detect rpath to OpenSSL libs

[tiran]

BPO	43466
Nosy	@warsaw, @gpshead, @vstinner, @tiran, @pablogsal, @miss-islington
PRs	bpo-43466: Add --with-openssl-rpath configure option (GH-24820) #24820 bpo-43466: Add -Wl,--exclude-libs to ssl #24989 bpo-43466: Unsupported static build hack #25002 bpo-43466: Use pkg-config with --static in the PY_UNSUPPORTED_OPENSSL_BUILD path in setup.py #25475 bpo-43466: Link with libz in PY_UNSUPPORTED_OPENSSL_BUILD path (GH-25587) #25587

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2021-03-29.17:46:19.796>
created_at = <Date 2021-03-10.23:14:03.328>
labels = ['expert-SSL', 'type-feature', 'expert-installation', '3.10', 'docs']
title = 'ssl/hashlib: Add configure option to set or auto-detect rpath to OpenSSL libs'
updated_at = <Date 2021-10-05.09:12:18.455>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-10-05.09:12:18.455>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2021-03-29.17:46:19.796>
closer = 'christian.heimes'
components = ['Documentation', 'Installation', 'SSL']
creation = <Date 2021-03-10.23:14:03.328>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 43466
keywords = ['patch']
message_count = 25.0
messages = ['388463', '388464', '388465', '388466', '388467', '388468', '388470', '388471', '388487', '388501', '388823', '388826', '388831', '388832', '388834', '389058', '389059', '389379', '389610', '389715', '391921', '403199', '403200', '403202', '403207']
nosy_count = 7.0
nosy_names = ['barry', 'gregory.p.smith', 'vstinner', 'christian.heimes', 'chaz6', 'pablogsal', 'miss-islington']
pr_nums = ['24820', '24989', '25002', '25475', '25587']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43466'
versions = ['Python 3.10']

[tiran]

Python's configure script has the option --with-openssl. It sets a path to a custom OpenSSL installation. Internally it provides OPENSSL_INCLUDES, OPENSSL_LIBS, and OPENSSL_LDFLAGS. The setup.py script turns the variables into include_dirs, library_dirs, and libraries arguments for _ssl and _hashlib extension modules.

However neither --with-openssl nor setup.py sets a custom runtime library path (rpath). This makes it confusing and hard for users to use a custom OpenSSL installation. They need to know that a) they have to take care of rpath on the first place, and b) how to set an rpath at compile or runtime. Without an rpath, the dynamic linker either fails to locate libssl/libcrypto or load system-provided shared libraries. Ticket bpo-34028 contains examples of user issues.

I propose to include a new option to make it easier for users to use a custom build of OpenSSL:

--with-openssl-rpath=<DIR|auto|no>

no (default): don't set an rpath
auto: auto-detect rpath from OPENSSL_LDFLAGS (--with-openssl or pkg-config)
DIR: set a custom rpath

The option will only affect the rpath of _ssl and _hashlib modules. The default value "no" is fully backwards compatible with 3.9 and earlier.

[pablogsal]

I have a suggestion (I am ok with also doing both). We should add also a flag that allows to statically compile openssl into the extension modules (if the .a are available) so there will be no shared object dependency. This allows us to have artifacts that are more self contained (regarding OpenSSL), partially eliminating the problem.

We could also go a step ahead and also hide the OpenSSL symbols from the dynamic table, so they don't collide if another shared object is already loaded in the future.

[tiran]

$ tar -xzf openssl-1.1.1j.tar.gz
$ pushd openssl-1.1.1j
$ ./config \
    --prefix=/home/heimes/dev/python/custom-openssl \
    --openssldir=\
        $(find /etc/ -name openssl.cnf -quit -printf "%h" 2>/dev/null)
$ make
$ make install_sw
$ popd

$ pushd cpython
$ ./configure \
    --with-openssl=/home/heimes/dev/python/custom-openssl \
    --with-openssl-rpath=auto
$ make
$ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffde07bc000)
        libssl.so.1.1 => /home/heimes/dev/python/custom-openssl/lib/libssl.so.1.1 (0x00007f937493a000)
        libcrypto.so.1.1 => /home/heimes/dev/python/custom-openssl/lib/libcrypto.so.1.1 (0x00007f937465a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9374608000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f937443d000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f9374436000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f93749ff000)
$ ./python
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1j  16 Feb 2021'
>>> import urllib.request
>>> r = urllib.request.urlopen("https://www.python.org")
                                   ^^^ No cert validation error!

[tiran]

I would rather not support static linking.

OpenSSL uses dynamic linking by default. Static linking is problematic for dynamic engine support. This is going to become an even bigger issue with OSSL providers in OpenSSL 3.0.0. I don't know yet how well OpenSSL 3.0.0 will support static linking.

Our macOS and Windows builds are now dynamically linked, too. At least Windows builds used to be statically linked.

Dynamic linked OpenSSL has a major benefit: It allows users to update OpenSSL without re-compiling Python.

[pablogsal]

OpenSSL uses dynamic linking by default.

As long as OpenSSL provide a .a file (they do) you can always static link. That is a decision of how you integrate the library.

Static linking is problematic for dynamic engine support.

Not sure I follow. What's the problem here? The advantage of static linking here will be to not have a dependency on the shared object, which can be quite beneficial.

Dynamic linked OpenSSL has a major benefit: It allows users to update OpenSSL without re-compiling Python.

Yeah, I agree. That's what should always be the default. But there are still advantages regarding static linking.

[pablogsal]

Note that I am not proposing to statically link the extensions into the core, I am proposing to statically link openssl into the extensions, so the symbols are in the text segment of the extension.

[tiran]

Not sure I follow. What's the problem here? The advantage of static linking here will be to not have a dependency on the shared object, which can be quite beneficial.

The problem is that some features are not baked into the .a files. They are always provided as shared libraries. This included OpenSSL engine extensions such as AFALG engine or external engines like p11-kit, OpenSC, or others. OpenSSL 3.0.0 moves some features into external OSSL provider libraries, for example legacy crypto algorithms. I have not figured out how much functionality we woud loose without engines and external OSSL providers. https://www.openssl.org/docs/manmaster/man3/OSSL_PROVIDER.html
# 3.0.0 alpha build:
$ find -name '*.so'
./engines-3/padlock.so
./engines-3/capi.so
./engines-3/afalg.so
./ossl-modules/fips.so
./ossl-modules/legacy.so
./libssl.so
./libcrypto.so

[pablogsal]

The problem is that some features are not baked into the .a files.

Oh, I see. So when using modern versions of openssl what shared objects do we expect to be in the NEEDED section? Right now, I see this in python3.8 in Ubuntu with OpenSSL 1.1.1:

❯ ldd /usr/lib/python3.8/lib-dynload/_ssl.cpython-38-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffcf71ef000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fe8782b6000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fe877deb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe877bcc000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe8777db000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe8775d7000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe87876e000)

[tiran]

It's very much the same for OpenSSL 3.0.0: libssl.so and libcrypto.so.

$ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffffa3cc000)
        libssl.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libssl.so.3 (0x00007f1ab0b66000)
        libcrypto.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libcrypto.so.3 (0x00007f1ab06b1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1ab065f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1ab0494000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f1ab048d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1ab0c55000)

The external engines and OSSL providers are external plugins. They are very much akin to Python's extension modules. OpenSSL loads them with dlopen(), dlsym()s an init function and finally calls the init function. It uses either RTLD_NOW or RTLD_NOW | RTLD_GLOBAL dlopen() flags.

The engines and OSSL providers depend on libcrypto.so. AFAIK this won't play will with static linking.

$ ldd ../multissl/openssl/3.0.0-alpha12/lib/engines-3/afalg.so 
        linux-vdso.so.1 (0x00007fffa417d000)
        libcrypto.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libcrypto.so.3 (0x00007fbcb3c75000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fbcb3c3e000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbcb3c1c000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fbcb3a51000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fbcb4133000)

$ ldd ../multissl/openssl/3.0.0-alpha12/lib/ossl-modules/legacy.so 
        linux-vdso.so.1 (0x00007ffd3ccc0000)
        libcrypto.so.3 => /home/heimes/dev/python/multissl/openssl/3.0.0-alpha12/lib/libcrypto.so.3 (0x00007f5524f36000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f5524eff000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5524edd000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f5524d12000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5525419000)

[pablogsal]

AFAIK this won't play will with static linking.

It should not be any problem as long as we expose the SSL symbols in the dynamic table of the extension (this happens by default). The only nuisance would be that users will still need those shared objects around so they can be dlopened by the extension.

[tiran]

Pablo,
in cc12888 you have regenerated configure with autoconf 2.71. The version is brand new and was released just 6 weeks ago. All my Linux machines have autoconf 2.69 from 2012 (!). Apparently 2.70 had some issues.

Would you mind if I regenerate configure with 2.69 and we stick with stable version for a little bit longer?

[pablogsal]

Would you mind if I regenerate configure with 2.69 and we stick with stable version for a little bit longer?

Please, go ahead. I need to remember then to use a docker container or something like that for the release then.

What's the problem that you are experiencing?

[vstinner]

Should the rpath be set on OpenSSL, on the Python ssl module, or on the Python executable?

no (default): don't set an rpath

If most users get it wrong, why not using "auto: auto-detect rpath from OPENSSL_LDFLAGS (--with-openssl or pkg-config)" by default?

The Fedora packaging disallows the usage of rpath.

All I need about rpath is that it can be dangerous in some cases, but I forgot about the details. For example, on Fedora, if OpenSSL is installed in the regular /usr/lib64/libssl.so directory, rpath must not be used. Otherwise, it can lead to crash, bugs or vulnerabilities.

Fedora, Python and rpath:

• https://pagure.io/packaging-committee/issue/886
• https://bugs.python.org/issue36659#msg340731

Tell me if you need more details about the issues when rpath is misued. But I need to ask my colleagues, I forgot the details.

[tiran]

It's a compromise.

The default settings for --with-openssl-rpath=no (--without-openssl-rpath) is backwards compatible with previous Python versions. The default behavor stays the same.

I don't want to set an rpath *unless* the user specifies that they want an rpath. I lack time and resources to verify that OPENSSL_LDFLAGS and OpenSSL's pkg-config files don't include any -L flags on all Linux, BSD, and macOS distributions. The new flag will make it more obvious to users that they may want an rpath, too.

$ ./configure --help
...
  --with-openssl=DIR      override root of the OpenSSL directory to DIR
  --with-openssl-rpath=[DIR|auto|no]
                          Set runtime library directory (rpath) for OpenSSL
                          libraries, no (default): don't set rpath, auto:
                          auto-detect rpath from --with-openssl and
                          pkg-config, DIR: set an explicit rpath
...

[vstinner]

The default settings for --with-openssl-rpath=no (--without-openssl-rpath) is backwards compatible with previous Python versions. The default behavor stays the same.

Ok. That makes sense.

[tiran]

New changeset 32eba61 by Christian Heimes in branch 'master':
bpo-43466: Add --with-openssl-rpath configure option (GH-24820)
32eba61

[tiran]

I'm leaving the ticket open as a reminder for me to update whatsnew.

[tiran]

#69177 adds -Wl,--exclude-libs just for libssl.a and libcrypto.a IFF support for -Wl,--exclude-libs,ALL is detected by configure. This puts the symbols from the OpenSSL archive files into the LOCAL segment of ELF binaries. The PR does not set -Wl,--exclude-libs,ALL because I like to keep behavior the same as with 3.9.

When OpenSSL is locally build with "no-shared -fPIC", then Python automatically builds a partially static-linked _ssl and _hashlib extension modules that do not pollute the global namespace:

$ ./config \
    --prefix=/home/heimes/dev/python/multissl/openssl/1.1.1j-static \
    --openssldir=/etc/pki/tls \
    no-shared -fPIC
...
$ ./configure --with-openssl=/home/heimes/dev/python/multissl/openssl/1.1.1j-static
$ make
$ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007fff8dbbc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa5a533d000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fa5a5172000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa5a56ac000)
$ readelf -Ws build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so | grep SSL_CTX_new
  5617: 0000000000072a90  1133 FUNC    LOCAL  DEFAULT   11 SSL_CTX_new

I deliberately did not update documentation with instructions for static linking. Static linking of OpenSSL has security and compatibility implications. I don't want to officially support it and deal with bug reports. -Wl,--exclude-libs just enables sane partial static-linking.

[miss-islington]

New changeset bacefbf by Christian Heimes in branch 'master':
bpo-43466: Unsupported static build hack (GH-25002)
bacefbf

[tiran]

There are now multiple ways to build Python with a custom OpenSSL build on Linux and BSD-like platforms:

1. Tools/ssl/multissltest.py
2. ./configure --with-openssl=/path/to/openssl --with-openssl-rpath=auto
3. undocumented hack from commit bacefbf
4. static extension modules with a custom "Modules/Setup.local" like this:

OPENSSL=/home/heimes/dev/python/multissl/openssl/1.1.1j-static
_ssl _ssl.c \
-I$(OPENSSL)/include -L$(OPENSSL)/lib \
-l:libssl.a -Wl,--exclude-libs,libssl.a \
-l:libcrypto.a -Wl,--exclude-libs,libcrypto.a
_hashlib _hashopenssl.c \
-I$(OPENSSL)/include -L$(OPENSSL)/lib \
-l:libcrypto.a -Wl,--exclude-libs,libcrypto.a

[tiran]

New changeset 5f87915 by Christian Heimes in branch 'master':
bpo-43466: Link with libz in PY_UNSUPPORTED_OPENSSL_BUILD path (GH-25587)
5f87915

[chaz6]

The new behaviour seems broken on my system.

$ PYTHON_VERSION=3.10.0

$ PKG_CONFIG_PATH="/home/chaz/.local/local/sqlite3/lib/pkgconfig:/home/chaz/.local/local/openssl3/lib64/pkgconfig"

$ LLVM_PROFDATA=/home/chaz/.local/local/clang+llvm/bin/llvm-profdata CC=clang CXX=clang++ ./configure --prefix=/home/chaz/.local/local/python-${PYTHON_VERSION} --enable-optimizations --enable-shared --enable-loadable-sqlite-extensions --with-openssl-rpath=auto

[..]

building '_ssl' extension
clang -pthread -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-instr-use=code.profclangd -I./Include/internal -I/home/chaz/.local/local/openssl-3.0.0/include -I./Include -I. -I/usr/local/include -I/home/chaz/Downloads/Python-3.10.0/Include -I/home/chaz/Downloads/Python-3.10.0 -c /home/chaz/Downloads/Python-3.10.0/Modules/_ssl.c -o build/temp.linux-x86_64-3.10/home/chaz/Downloads/Python-3.10.0/Modules/_ssl.o
warning: no profile data available for file "_ssl.c" [-Wprofile-instr-unprofiled]
clang -pthread -shared build/temp.linux-x86_64-3.10/home/chaz/Downloads/Python-3.10.0/Modules/_ssl.o -L/home/chaz/.local/local/openssl-3.0.0/lib64 -L. -L/usr/local/lib -R/home/chaz/.local/local/openssl-3.0.0/lib64 -lssl -lcrypto -o build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so
clang -pthread -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-instr-use=code.profclangd -I./Include/internal -I/home/chaz/.local/local/openssl-3.0.0/include -I./Include -I. -I/usr/local/include -I/home/chaz/Downloads/Python-3.10.0/Include -I/home/chaz/Downloads/Python-3.10.0 -c /home/chaz/Downloads/Python-3.10.0/Modules/_hashopenssl.c -o build/temp.linux-x86_64-3.10/home/chaz/Downloads/Python-3.10.0/Modules/_hashopenssl.o
warning: no profile data available for file "_hashopenssl.c" [-Wprofile-instr-unprofiled]
clang -pthread -shared build/temp.linux-x86_64-3.10/home/chaz/Downloads/Python-3.10.0/Modules/_hashopenssl.o -L/home/chaz/.local/local/openssl-3.0.0/lib64 -L. -L/usr/local/lib -R/home/chaz/.local/local/openssl-3.0.0/lib64 -lssl -lcrypto -o build/lib.linux-x86_64-3.10/_hashlib.cpython-310-x86_64-linux-gnu.so
*** WARNING: renaming "_ssl" since importing it failed: libssl.so.3: cannot open shared object file: No such file or directory
*** WARNING: renaming "_hashlib" since importing it failed: libssl.so.3: cannot open shared object file: No such file or directory

$ ldd /home/chaz/Downloads/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu_failed.so
        linux-vdso.so.1 =>  (0x00007fffc99f4000)
        libssl.so.3 => not found
        libcrypto.so.3 => not found
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faa70802000)
        libc.so.6 => /lib64/libc.so.6 (0x00007faa70434000)
        /lib64/ld-linux-x86-64.so.2 (0x00007faa70a1e000)

$ readelf -d /home/chaz/Downloads/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu_failed.so | grep PATH

It appears that RPATH/RUNPATH is not getting set.

[chaz6]

If I set LDFLAGS as with previous versions, it works as expected. I do not know if it is related, but I noticed that by default rpath does not get set for the files in DESTDIR/bin/* (e.g. python3.10 or pip3.10) hence the inclusion of /home/chaz/.local/local/python-${PYTHON_VERSION}/lib in the rpath.

LDFLAGS="-Wl,-rpath,'/home/chaz/.local/local/openssl3/lib64:/home/chaz/.local/local/sqlite3/lib:/home/chaz/.local/local/python-'${PYTHON_VERSION}'/lib' -Wl,--enable-new-dtags"

[tiran]

Chris, this bug report is closed. Please open a new bug report.

I think it's a bug in distutils. UnixCCompiler.runtime_library_dir_option() may not support clang and add wrong option "-R/home/chaz/.local/local/openssl-3.0.0/lib64".

[tiran]

Chris, please try the fix from bpo-45371.