configure incorrectly ignores pkg-config information for libffi and Tcl/Tk in 3.10

[ned-deily]

BPO	45350
Nosy	@tiran, @ned-deily, @pablogsal, @thesamesam
PRs	[3.10] bpo-45350: Rerun autoreconf with the pkg-config macros #28707 bpo-45350: Rerun autoreconf with the pkg-config macros #28708

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pablogsal'
closed_at = <Date 2021-10-03.23:47:33.351>
created_at = <Date 2021-10-03.04:13:54.679>
labels = ['build', '3.10', '3.11']
title = 'configure incorrectly ignores pkg-config information for libffi and Tcl/Tk in 3.10'
updated_at = <Date 2021-10-29.16:13:11.880>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2021-10-29.16:13:11.880>
actor = 'thesamesam'
assignee = 'pablogsal'
closed = True
closed_date = <Date 2021-10-03.23:47:33.351>
closer = 'pablogsal'
components = ['Build']
creation = <Date 2021-10-03.04:13:54.679>
creator = 'ned.deily'
dependencies = []
files = []
hgrepos = []
issue_num = 45350
keywords = ['patch']
message_count = 16.0
messages = ['403076', '403081', '403082', '403088', '403093', '403094', '403095', '403096', '403098', '403099', '403100', '403102', '403116', '403117', '403177', '405018']
nosy_count = 4.0
nosy_names = ['christian.heimes', 'ned.deily', 'pablogsal', 'thesamesam']
pr_nums = ['28707', '28708']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45350'
versions = ['Python 3.10', 'Python 3.11']

[ned-deily]

./configure supports using the system or third-party-supplied pkg-config utility to find or override the default location of header and library files when building a few C extensions in the standard library, namely from OpenSSL for the _ssl module, libffi for ctypes, and, new in 3.10, Tcl/Tk for _tkinter (bpo-42603). However, currently for 3.10.0, pkg-config usage is broken for libffi and Tcl/Tk (but not OpenSSL). When running ./configure, there is an unexpected warning that is easily overlooked:

[...]
checking for --with-libs... no
./configure: line 10545: PKG_PROG_PKG_CONFIG: command not found
checking for --with-system-expat... no
[...]

PKG_PROG_PKG_CONFIG is a macro provided by GNU Autotools that is supposed to be in aclocal.m4. Unfortunately, it appears to have been inadvertently deleted in 2fc857a (PR 25860) probably due to an autoconf version mismatch. The net effect is that the configure variable PKG_CONFIG, the location of the pkg-config utility, is undefined so tests in configure for the location of libffi and of Tcl and Tk do not take into account any pkg-config info and use any default locations (i.e. /usr/include). For most builds, that likely still produces the desired results. But it won't if a builder is trying to override these locations or is building on a platform with different default or is using a third-party package manager and pkg-config to supply libraries. Note, the _ssl module builds are not affected by this problem as the AX_CHECK_OPENSSL macro in aclocal.m4 does not depend on PKG_PROG_PKG_CONFIG to find pkg-config.

It appears that the problem can be worked around by explicitly setting the PKG_CONFIG build variable when invoking configure, something like:

./configure [...] PKG_CONFIG=$(which pkg-config)

But the PR 25860 changes to aclocal.a4 should be carefully reviewed and the pkg-config related deletes restored; there might be other problems, too. This is not the first time we've been caught up by unexpected autoconf changes and, as is clear here, it is too easy for them to go unnoticied. Perhaps we should try to figure out how to reduce those risks.

[pablogsal]

it appears to have been inadvertently deleted in 2fc857a (PR 25860) probably due to an autoconf version mismatch.

Since then, I have updated the autotools version to the old one in #28151
so this should have been fixed by that.
What I am missing?

[pablogsal]

Regarding preventing this, I have added a check so the autotools version doesn't get updated incorrectly:

#28152

[ned-deily]

The updates to aclocal.m4 are generated by the aclocal tool of the GNU build system. That, and other things including autoconf, gets run by autoreconf. Running autoconf by itself is not sufficient.

https://devguide.python.org/setup/#regenerate-configure

https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/autoreconf-Invocation.html

[pablogsal]

For regeneration I was using:

https://github.com/tiran/cpython_autoconf

as instructed by Christian.

[pablogsal]

This runs:

"autoreconf -ivf"

with the correct version of autotools required by the openssl changes

[pablogsal]

I can confirm that running autoreconf with Christian's version doesn't change the git tree:

❯ docker run -vpwd:/src tiran/cpython_autoconf
Rebuilding configure script
autoreconf: Entering directory .' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal --force autoreconf: configure.ac: tracing autoreconf: configure.ac: not using Libtool autoreconf: running: /usr/bin/autoconf --force autoreconf: running: /usr/bin/autoheader --force autoreconf: configure.ac: not using Automake autoreconf: Leaving directory .'
Done

3.10 on  3.10 [$]  pyenv 3.9.1 took 6s
❯ git status
On branch 3.10
Your branch is up to date with 'upstream/3.10'.

nothing to commit, working tree clean

[pablogsal]

Ah, the problem is that pkgconfig is *not* installed in that DOckerfile. I can confirm that installing pkg-config and rerunning works.

[pablogsal]

Created a PR for this:

pablogsal/cpython_autoconf#1

I think we should move this dockerfile to the CPython repo and update the devguide.

[ned-deily]

+1, I wasn't aware of the existence of that.

[pablogsal]

Something seems wrong with the openssl detection when the PKG_PROG_PKG_CONFIG is active :(

Christian, could you please take a look? I would like to get this ready for the release

[tiran]

I have updated the container at https://quay.io/repository/tiran/cpython_autoconf to include pkg-config.

[pablogsal]

New changeset f146ca3 by Pablo Galindo Salgado in branch '3.10':
bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28707)
f146ca3

[pablogsal]

New changeset a25dcae by Pablo Galindo Salgado in branch 'main':
bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28708)
a25dcae

[pablogsal]

New changeset 2c47b80 by Pablo Galindo (Pablo Galindo Salgado) in branch '3.10':
bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28707)
2c47b80

[ned-deily]

Note that we have since discovered that the fix for this problem inadvertently did not make it into the 3.10.0 release. It is in the current 3.10 branch and will be in 3.10.1, the first bugfix release for 3.10.