Port module setup to PY_STDLIB_MOD() macro and addext()

[tiran]

BPO	45847
Nosy	@tiran, @ned-deily, @pablogsal, @miss-islington, @erlend-aasland, @arhadthedev
PRs	bpo-45847: Port builtin hashlib extensions to PY_STDLIB_MOD (GH-29642) #29642 bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644) #29644 bpo-45847: Port test modules to PY_STDLIB_MOD (GH-29660) #29660 bpo-45847: Port grp, spwd, termios, resource, syslog to PY_STDLIB_MOD (GH-29668) #29668 bpo-45847: Port codecs and unicodedata to PY_STDLIB_MOD (GH-29685) #29685 bpo-45847: Port array, _contextvars, math, and cmath to PY_STDLIB_MOD_SIMPLE (GH-29688) #29688 bpo-45847: Port _bisect, _heapq, _json, _pickle, _random, and _zoneinfo to PY_STDLIB_MOD_SIMPLE #29689 bpo-45847: Port _lfprof, _opcode, _asyncio, _queue, _statistics, and _typing to PY_STDLIB_MOD_SIMPLE #29690 bpo-45847: Port fcntl to Py_STDLIB_MOD (GH-29696) #29696 bpo-45847: Various PY_STDLIB_MOD cleanups (GH-29697) #29697 bpo-45847: Port nis module to PY_STDLIB_MOD (GH-29699) #29699 bpo-45847: Port compression libs to PY_STDLIB_MOD (GH-29702) #29702 bpo-45847: Port mmap, select, and _xxsubinterpreters to Py_STDLIB_MOD (GH-29703) #29703 bpo-45847: Port audioop, _csv, and _posixsubprocess to PY_STDLIB_MOD_SIMPLE #29705 bpo-45847: port _struct to PY_STDLIB_MOD (GH-29706) #29706 bpo-45847: Port xxlimited and xxlimited_35 to PY_STDLIB_MOD (GH-29707) #29707 bpo-45847: Port _socket to PY_STDLIB_MOD (GH-29713) #29713 bpo-45847: Fix xxlimited and xxlimited_35 build conditions (GH-29715) #29715 bpo-45847: Port _gdbm to PY_STDLIB_MOD (GH-29720) #29720 bpo-45847: Port _crypt to PY_STDLIB_MOD (GH-29725) #29725 bpo-45847: Port _ssl and _hashlib to PY_STDLIB_MOD (GH-29727) #29727 bpo-45847: Port _posixshmem to PY_STDLIB_MOD (GH-29738) #29738 bpo-45847: Port _uuid to PY_STDLIB_MOD (GH-29741) #29741 bpo-45847: Fix _crypt detection on Ubuntu (GH-29743) #29743 bpo-45847: Port _ctypes partly to PY_STDLIB_MOD (GH-29747) #29747 bpo-45847: Port _multiprocessing to PY_STDLIB_MOD (GH-29768) #29768 bpo-45847: Make socket module conditional (GH-29769) #29769 bpo-45847: Fix _multiprocessing on platforms without sem_open (GH-29833) #29833 bpo-45847: PY_STDLIB_MOD_SIMPLE now checks py_stdlib_not_available (GH-29844) #29844 bpo-45847: Update whatsnew and add place holder entries for missing extensions (GH-29914) #29914 bpo-45847: Fix uuid detection on macOS (GH-29946) #29946 bpo-45847: Don't override user overrides for CFLAGS/LIBS (GH-29967) #29967 bpo-45847: Revert Port _ctypes partly to PY_STDLIB_MOD (GH-29747) (GH-29969) #29969 bpo-45847: Port _tkinter to PY_STDLIB_MOD #31698 gh-90005: Port _ctypes to PY_STDLIB_MOD #32229 Fix Contributed ... in bpo-bpo-45847 typo #32299 bpo-45847: Adapt macOS installer build to use new tkinter configure vars #32328

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-19.16:56:30.299>
labels = ['type-feature', 'build', '3.11']
title = 'Port module setup to PY_STDLIB_MOD() macro and addext()'
updated_at = <Date 2022-04-05.04:48:03.902>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-04-05.04:48:03.902>
actor = 'Anthony Sottile'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-11-19.16:56:30.299>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 45847
keywords = ['patch']
message_count = 61.0
messages = ['406597', '406618', '406646', '406662', '406703', '406714', '406746', '406751', '406754', '406755', '406763', '406771', '406777', '406779', '406781', '406782', '406783', '406784', '406785', '406788', '406801', '406822', '406823', '406842', '406890', '406891', '406892', '406905', '406913', '406914', '407077', '407199', '407249', '407255', '407274', '407287', '407654', '407668', '407861', '407949', '407958', '407959', '407962', '407981', '408008', '414642', '414645', '414648', '414649', '414656', '416425', '416565', '416566', '416567', '416568', '416569', '416570', '416575', '416600', '416712', '416750']
nosy_count = 6.0
nosy_names = ['christian.heimes', 'ned.deily', 'pablogsal', 'miss-islington', 'erlendaasland', 'arhadthedev']
pr_nums = ['29642', '29644', '29660', '29668', '29685', '29688', '29689', '29690', '29696', '29697', '29699', '29702', '29703', '29705', '29706', '29707', '29713', '29715', '29720', '29725', '29727', '29738', '29741', '29743', '29747', '29768', '29769', '29833', '29844', '29914', '29946', '29967', '29969', '31698', '32229', '32299', '32328']
priority = None
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45847'
versions = ['Python 3.11']

Linked PRs

• gh-90005: Don't link with libbsd if not needed #105236
• [3.11] gh-90005: Don't link with libbsd if not needed (#105236) #105359
• [3.12] gh-90005: Don't link with libbsd if not needed (#105236) #105360

[tiran]

bpo-45573 added the PY_STDLIB_MOD() autoconf macro, Modules/Setup.stdlib.in template, and setup.py helper method addext().

The macro sets up

• MODULE_{NAME}_TRUE/FALSE conditional
• MODULE_{NAME} variable with values yes, disabled, missing, or n/a
• MODULE_{NAME}_CFLAGS
• MODULE_{NAME}_LDFLAGS

Additionally there is a MODULE_{NAME}_DEPS variable with dependency information.

The Modules/Setup.stdlib.in template and addext() method consume the variables and set up extension compilation and linking. There is no need to look for headers or libraries in setup.py any more.

Let's port all modules to the new approach.

[tiran]

New changeset c8c21bd by Christian Heimes in branch 'main':
bpo-45847: Port builtin hashlib extensions to PY_STDLIB_MOD (GH-29642)
c8c21bd

[tiran]

New changeset 5596909 by Christian Heimes in branch 'main':
bpo-45847: Port _scproxy to PY_STDLIB_MOD (GH-29644)
5596909

[tiran]

New changeset f36c69a by Christian Heimes in branch 'main':
bpo-45847: Port test modules to PY_STDLIB_MOD (GH-29660)
f36c69a

[tiran]

New changeset f201d26 by Christian Heimes in branch 'main':
bpo-45847: Port grp, spwd, termios, resource, syslog to PY_STDLIB_MOD (GH-29668)
f201d26

[tiran]

New changeset 2afa1a1 by Christian Heimes in branch 'main':
bpo-45847: Port codecs and unicodedata to PY_STDLIB_MOD (GH-29685)
2afa1a1

[tiran]

New changeset 133c65a by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port array, _contextvars, math, and cmath to PY_STDLIB_MOD_SIMPLE (GH-29688)
133c65a

[miss-islington]

New changeset 718cee0 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _bisect, _heapq, _json, _pickle, _random, and _zoneinfo to PY_STDLIB_MOD_SIMPLE (GH-29689)
718cee0

[erlend-aasland]

45 down, 35 to go.

[miss-islington]

New changeset 39f7d2f by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _lfprof, _opcode, _asyncio, _queue, _statistics, and _typing to PY_STDLIB_MOD_SIMPLE (GH-29690)
39f7d2f

[tiran]

New changeset 29699a2 by Christian Heimes in branch 'main':
bpo-45847: Various PY_STDLIB_MOD cleanups (GH-29697)
29699a2

[tiran]

New changeset 5b946ca by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port fcntl to Py_STDLIB_MOD (GH-29696)
5b946ca

[tiran]

New changeset b451673 by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port mmap, select, and _xxsubinterpreters to Py_STDLIB_MOD (GH-29703)
b451673

[tiran]

New changeset d9cedab by Christian Heimes in branch 'main':
bpo-45847: Port compression libs to PY_STDLIB_MOD (GH-29702)
d9cedab

[tiran]

New changeset c6dec7e by Christian Heimes in branch 'main':
bpo-45847: Port nis module to PY_STDLIB_MOD (GH-29699)
c6dec7e

[tiran]

Not converted yet:

readline
_curses
_curses_panel
_crypt
_socket
_ssl
_hashlib
_dbm
_gdbm
_ctypes
_multiprocessing
_posixshmem
_tkinter
_uuid
xxlimited
xxlimited_35

[erlend-aasland]

_socket coming up...

[miss-islington]

New changeset eee683c by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port audioop, _csv, and _posixsubprocess to PY_STDLIB_MOD_SIMPLE (GH-29705)
eee683c

[tiran]

_crypt is powered by pkgconf libcrypt or libxcrypt on Linux. On my system and on Debian, libcrypt.pc is a symlink to libxcrypt.pc. Linux has crypt() and/or crypt_r() in <crypt.h>. On FreeBSD crypt_r() is in <unistd.h> and libc.

_uuid needs similar special handling. It's <uuid/uuid.h>, -luuid, and uuid.pc on Linux. On BSD it's <uuid.h> and symbols are in libc.

_readline uses either libreadline (readline.pc) or libeditline (libeditline.pc). It also needs special handling of tinfo and termcap on some platforms.

ndbm, gdbm, and libdb have no pkgconf providers. My PR #73720 has some code to detect them in configure.

[tiran]

New changeset 0e1c2f3 by Christian Heimes in branch 'main':
bpo-45847: port _struct to PY_STDLIB_MOD (GH-29706)
0e1c2f3

[tiran]

New changeset 2dc7d3d by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port xxlimited and xxlimited_35 to PY_STDLIB_MOD (GH-29707)
2dc7d3d

[tiran]

New changeset 758a23d by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Fix xxlimited and xxlimited_35 build conditions (GH-29715)
758a23d

[tiran]

New changeset d82f2ca by Erlend Egeberg Aasland in branch 'main':
bpo-45847: Port _socket to PY_STDLIB_MOD (GH-29713)
d82f2ca

[tiran]

We are down to 9 missing modules:

readline
_curses
_curses_panel
_dbm
_ctypes
_multiprocessing
_posixshmem
_tkinter
_uuid

[erlend-aasland]

Looks like we're done! Thanks for the work with the remaining modules, Christian! I believe we can close this issue now.

[erlend-aasland]

See #93939 for the next step :)