How to detect OpenSSL multilib suffix for linking?

[native-api]

I'm a maintainer of the Pyenv project. Our users are having problems linking CPython against a custom installation of OpenSSL (see e.g. pyenv/pyenv#3497).

To find OpenSSL, CPython's Configure relies on ax_check_openssl.m4 which (as of this writing), given an explicit prefix, hardcodes the library path to <prefix>/lib.
However, since 3.0.0, a source install of OpenSSL always adds a predefined multilib suffix to "lib".

---

Looking for ways to fix this problem, I found out that there seems to be no way to reliably determine which multilib suffix a particular installation of OpenSSL has been configured with. All files that contain it are themselves located under a directory with that suffix:

$~/lib/openssl# grep -r lib64 .
Binary file ./lib64/libcrypto.so.4 matches
./lib64/pkgconfig/openssl.pc:libdir=${exec_prefix}/lib64
./lib64/pkgconfig/libssl.pc:libdir=${exec_prefix}/lib64
./lib64/pkgconfig/libcrypto.pc:libdir=${exec_prefix}/lib64
Binary file ./lib64/libcrypto.a matches
./lib64/cmake/OpenSSL/OpenSSLConfig.cmake:set(OPENSSL_LIBRARY_DIR "${_ossl_prefix}/lib64")
./lib64/cmake/OpenSSL/OpenSSLConfig.cmake:set(OPENSSL_MODULES_DIR "${_ossl_prefix}/lib64/ossl-modules")

• Getting the local multilib suffix for the current architecture (e.g. with gcc -print-multi-directory) doesn't help here because OpenSSL doesn't use that but rather uses an independent hardcoded value. That value is only present in <build>/configdata.pm which is not available after installation.
• Patching ax_check_openssl.m4 to check for existence of $ssldir/lib* seems unreliable: e.g. there may be multiple matches and/or unrelated matches.

---

What is the intended way here?

[LeonardoSanBenitez]

The postfix is deliberate and --libdir is the only supported lever over it. It became unconditional in 3.0 with 74b7f33; before that it was applied only if $prefix/lib<multilib> already existed on the build machine, which is why ax_check_openssl.m4's hardcoded <prefix>/lib worked against 1.1.1 installs into a fresh prefix and stopped working with 3.0. When that broke people (#16244), a dedicated override was proposed in #16246 and OTC turned it down with "we do not need this as --libdir is sufficient workaround", and the outcome was the INSTALL.md/CHANGES wording you found. So there is nothing else to query and nothing else coming.

One thing worth knowing before you write any detection for it: the value is a per-target constant and not always a bare suffix. linux-x86_64 gives lib64, linux-x32 gives libx32, VC-WIN64A gives lib-x64, but solaris64-* give lib/64 and linux32-s390x gives lib/highgprs, i.e. a nested directory. No darwin or BSD target sets it at all, which is presumably why only your Linux users hit this.

For CPython I think you can get what you want out of flags that already exist, without patching the macro. ax_check_openssl.m4 only consults pkg-config in the branch where --with-openssl was not given (serial 11, lines 51-68; the given branch goes straight to -L$ssldir/lib on line 81, and the copy CPython ships is the same). PKG_CONFIG_PATH takes a list, and pkg-config matches the file openssl.pc rather than a directory pattern, so you can enumerate candidates without the ambiguity that made you reject <prefix>/lib* — entries that do not exist are skipped.

Both runs below are CPython 3.14.6 configured against a 3.5 tree installed at /opt/openssl-3.5, on debian bookworm which also has the distro's 3.0 libssl-dev, so it is one linux/glibc/x86_64 data point:

# what pyenv does today
./configure --with-openssl=/opt/openssl-3.5
#   OPENSSL_LDFLAGS=-L/opt/openssl-3.5/lib        <- that directory does not exist
#   gcc -Wl,-t shows the linker opening /usr/lib/x86_64-linux-gnu/libcrypto.so,
#   while the headers come from the prefix. Every OpenSSL check still prints "yes".

# candidates, suffix never known in advance
PKG_CONFIG_PATH=/opt/openssl-3.5/lib/pkgconfig:/opt/openssl-3.5/lib64/pkgconfig:/opt/openssl-3.5/lib32/pkgconfig:/opt/openssl-3.5/libx32/pkgconfig \
  ./configure --with-openssl-rpath=auto
#   MODULE__SSL_CFLAGS=-I/opt/openssl-3.5/include
#   MODULE__SSL_LDFLAGS=-L/opt/openssl-3.5/lib64 -Wl,--enable-new-dtags,-rpath=/opt/openssl-3.5/lib64 -lssl -lcrypto

The first is the silent wrong-library case behind the undefined symbol in pyenv#3497: headers 3.5.8-dev, library 3.0.20, configure happy.

If you would rather ask the installation itself, <prefix>/bin/openssl info -modulesdir prints <libdir>/ossl-modules (MODULESDIR = $(libdir)/ossl-modules in Configurations/unix-Makefile.tmpl) and openssl-info(1) is explicitly meant for scripts. Two caveats: it exists from 3.0 on, 1.1.1 has no info command; and the value describes the libcrypto that got loaded rather than the binary you ran. I checked that by running the system 3.0 openssl against the 3.5 libcrypto, and it prints the 3.5 tree's directory. So on a machine where the prefix's libdir is not on the loader path you can get the system's answer or no answer at all, and it is worth checking that what comes back starts with the prefix you asked about.

For the OpenSSL builds you control, --libdir=lib is still the simplest thing.

Disclaimer: these messages are being posted by a claude code agent. I, Leonardo, a human, am only proving loose supervision, and I'm not verifiyng every message. If you believe something is wrong, feel free to contact me here or in linkedin or via email.

[native-api]

Okay, this confirms that there's no supported way.
Would you accept a pull request that would add autodetection of the local suffix in Linux as part of detecting the current architecture?

[LeonardoSanBenitez]

Worth saying plainly first: I'm not a committer and can't tell you whether the project would take it — I'm someone reading the source and running builds, and fairly new to this codebase, so nothing below is a project position. CONTRIBUTING.md's own route for that question is to open an issue before starting work ("Someone may be already working on the same thing, or there may be special reasons why a feature is not implemented"); a Q&A thread can't produce a commitment from anybody.

What I can give you is the record, and I'd read it before spending the time. Your proposal was already made by a committer, in the thread that produced the docs wording: in #16244 t8m wrote "Perhaps we could somehow detect that the system is using a multilib layout in the Red Hat/Fedora way vs. the Debian way?", and opened #16246 (a --multilib option) the same day. Four days later both got an OTC answer — "This should be treated as a documentation problem" and "we do not need this as --libdir is sufficient workaround". And what 74b7f33 removed in 3.0 was itself an autodetect: LIBDIR fell back to plain lib unless $prefix/lib<multilib> already existed, and it went out because "build results should not depend on the build machine root filesystem layout". So a new autodetect reopens a decision that was made on purpose, and the patch is the easy half.

My opinion, labelled as one and worth what a newcomer's opinion is worth: default-on autodetection looks like a hard sell against that reasoning, and an opt-in switch is #16246 again. What has changed since 2021 is that you can point at concrete downstream breakage, which those threads didn't have — that seems more likely to move it than a diff.

If you do write it, the detection isn't easy either, because "the local suffix" isn't one signal. Measured on x86_64 containers this week: -d /usr/lib64 says yes on Debian 12 (it exists and holds only the dynamic loader) and yes on Arch (it's a symlink to lib), and both of those are lib distros. Two probes did track the convention on all five I tried. One is whether <prefix>/lib64 exists for prefix /usr/local — the test 74b7f33 deleted — absent on Debian, Arch and Alpine, present on Fedora and openSUSE; its catch is the one from #16121, that a custom prefix which doesn't exist yet answers nothing. The other is gcc -print-multi-os-directory: ../lib on Debian, Arch and Alpine, ../lib64 on Fedora and openSUSE. That's a different flag from the -print-multi-directory you ruled out, which returns . on all five. For comparison, CMake's GNUInstallDirs doesn't touch the filesystem: /etc/debian_version, /etc/arch-release, /etc/alpine-release, otherwise lib64 if the target is 64-bit, and the whole block is skipped when cross-compiling.

Mechanically it is small, and there's a hook that fits your framing exactly. On master today, the arms in util/perl/OpenSSL/config.pm return a hash that Configure merges straight into %config (Configure:1344), and unix-Makefile.tmpl:321 already prefers $config{libdir} over lib$target{multilib} — so returning libdir => ... from the x86_64-.*-linux arm, which already runs the compiler for its ILP32 check, is a few lines. Two things a reviewer would poke at: --libdir is parsed into that same key before the merge, so the guesser would silently clobber an explicit --libdir unless it's guarded; and the guesser only runs when no target was named, so ./Configure would autodetect while ./Configure linux-x86_64 kept the constant.

Two things hold either way. It would be master-only — it contradicts the documented behaviour in INSTALL.md rather than conforming to it, so as I read the stable-release-updates policy it isn't a backportable bug fix — which means every 3.x/4.x install already out there keeps its current layout and pyenv still has to cope with those. And #31845 is open right now, adding multilib => "64" to linux-ppc64le and linux-aarch64; if it lands this reaches your arm64 users too, and that PR is where the Debian-versus-lib64 question came up most recently.

Disclaimer: these messages are being posted by a claude code agent. I, Leonardo, a human, am only proving loose supervision, and I'm not verifiyng every message. If you believe something is wrong, feel free to contact me here or in linkedin or via email.

[idrassi]

@LeonardoSanBenitez, please don't post unverified output from an AI agent as technical guidance here. Saying that a bot wrote the text is not a substitute for reviewing it. The person posting it remains responsible, and several claims above are wrong or lead to unsafe fallback behavior.

@native-api, I checked the OpenSSL history and the CPython 3.14.6 configure code. I don't think a prefix alone can reliably identify the library directory of an existing OpenSSL installation.

On Unix, a target's multilib value only supplies the default used when --libdir was not given. The build may use any relative or absolute --libdir, and one prefix may hold several ABI variants. A compiler or distribution check can suggest a default for a new build. It can't tell us how an existing installation was configured.

Commit 74b7f33 removed the old check for an existing directory because it made builds depend on the build machine file system. One earlier claim needs correcting: issue #16244 did report downstream breakage, namely a libcurl link failure. PR #16246 proposed a manual --multilib option, not distribution detection, and was closed because --libdir already provides that override. This explains the present choice, but doesn't rule out discussing a better default for future builds.

What concerns me more is CPython accepting a different OpenSSL from the one the user requested. Its vendored AX_CHECK_OPENSSL finds headers under <prefix>/include, then sets:

OPENSSL_LDFLAGS=-L<prefix>/lib
OPENSSL_LIBS=-lssl -lcrypto

An -L option adds a search directory. It doesn't confine the linker to that directory, and this macro places it after any existing LDFLAGS. If <prefix>/lib is absent, incomplete, or lower in the search order, the linker may combine the requested headers with system libraries. The first check can pass, configure can finish, and the problem may appear only when _ssl or _hashlib is imported. It may also appear to work while quietly using the wrong OpenSSL.

The runtime loader then makes another choice. The selected libcrypto also supplies the OpenSSL configuration and its provider and module paths. With OpenSSL 3, this can affect provider loading and the FIPS setup. Passing --with-openssl-rpath doesn't fix the original library choice, it only adds a runtime search path.

In my opinion, an explicit dependency root must never fall back silently to another installation. CPython should stop when it can't find matching libssl and libcrypto libraries, and it should accept a separate library directory or specifically selected package metadata. The same lib versus lib64 problem was reported in CPython #121992. CPython own instructions avoid it by building OpenSSL with --libdir=lib.

I don't see listing guessed directories in PKG_CONFIG_PATH as a general solution. CPython uses that route only when --with-openssl is omitted. The variable adds to the normal search path, so a failed guess can still select the system openssl.pc, while several matches are resolved by taking the first one. A caller should first find one specific openssl.pc, query it separately with an empty PKG_CONFIG_PATH and a restricted PKG_CONFIG_LIBDIR, then check its version, paths, and actual library files. Otherwise, it should ask for the library directory explicitly.

The <prefix>/bin/openssl info -modulesdir idea is only diagnostic. Its answer comes from the libcrypto selected by the runtime loader, not necessarily from the installation containing that executable. It also can't provide general discovery during cross compilation.

So we should keep the two issues separate:

• For the current consumer problem, CPython and pyenv should select exact metadata or an explicit library directory, and stop on ambiguity or fallback. When the caller controls the OpenSSL build, it can avoid discovery by passing the desired --libdir.
• A new OpenSSL default for Linux can be discussed on its own. A compiler query may help choose that default for future builds, but it can't tell where an existing OpenSSL installation placed its libraries.

[native-api]

Thank you for the detailed answer! In the current situation, I have decided to have Pyenv replace user-provided --with-openssl with compiler flags, detecting the prefix in some ad-hoc manner. Will probably search for libssl* under $prefix/lib* (will check the OpenSSL source for a specific pattern that's guaranteed to work) and fail the build if multiple are found (as this workaround is for user-installed source builds, that's very unlikely).

[native-api]

Looking at how other 3rd-party libraries handle this issue, I see that other libraries don't deal with multilib/multiarch altogether. Instead, they accept configure options for a number of install directories, including lib. Since multilib/multiarch is imposed by a distro rather than a library, it's the job of the distro's maintainers to provide those to the build in their packaging.

As such, it seems that the best course of action for OpenSSL is to drop the multilib machinery altogether as seems to be the SOP in the industry.

At the very least, it's logical to do so if configure detects distro-provided multilib/multiarch machinery to be present.

[native-api]

At the very least, it's logical to do so if the build detects distro-provided multilib/multiarch machinery to be present.

To make this fact explicit and avoid nasty surprises for downstream users, I suggest to rewrite corresponding values in applicable OpenSSL build configurations as "detect|<default>".

[LeonardoSanBenitez]

Hi @idrassi, thanks for the feedback, I'll be more careful with LLM-aided replies.

I completely agree that an explicit dependency root must never fall back silently to another installation.
Do you think any part of the documentation or error messages could be improved, so as to decrease this type of problem/confusion/question in the future? I would be glad to try to contribute there.

rewrite corresponding values in applicable OpenSSL build configurations as detect|<default>

Sounds like a good idea to me :)