The "m" ABI flag of SOABI for pymalloc is no longer needed

[vstinner]

BPO	36707
Nosy	@vstinner, @ned-deily, @serhiy-storchaka, @koobs, @hroncok, @miss-islington
PRs	bpo-36707: Remove "m" flag from SOABI #12931 bpo-36707: Document "m" removed from sys.abiflags #14090 [3.8] bpo-36707: Document "m" removal from sys.abiflags (GH-14090) #14097

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2019-06-15.09:36:07.714>
created_at = <Date 2019-04-24.00:27:25.531>
labels = ['deferred-blocker', 'build', '3.8']
title = 'The "m" ABI flag of SOABI for pymalloc is no longer needed'
updated_at = <Date 2019-06-15.09:39:47.645>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-06-15.09:39:47.645>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2019-06-15.09:36:07.714>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-04-24.00:27:25.531>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36707
keywords = ['patch']
message_count = 21.0
messages = ['340748', '340749', '340757', '340766', '340773', '341097', '341099', '341100', '341121', '341835', '344478', '344483', '344484', '344521', '344524', '344526', '344528', '345650', '345651', '345666', '345667']
nosy_count = 6.0
nosy_names = ['vstinner', 'ned.deily', 'serhiy.storchaka', 'koobs', 'hroncok', 'miss-islington']
pr_nums = ['12931', '14090', '14097']
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36707'
versions = ['Python 3.8']

[vstinner]

In Python 2.7, the WITH_PYMALLOC define (defined by ./configure --with-pymalloc which is the default) modified the Python API. For example, PyObject_MALLOC() is a macro replaced with PyObject_Malloc() with pymalloc but replaced with PyMem_MALLOC() without pymalloc.

In Python 3.8, it's no longer the case, PyObject_MALLOC is no longer modified by WITH_PYMALLOC, it's always an alias to PyObject_Malloc() for backward compatibility.

#define PyObject_MALLOC   PyObject_Malloc

More generally, WITH_PYMALLOC has no impact on the Python headers nor on the ABI in any way. Memory allocators can be switched at runtime using PYTHONMALLOC environment variable and -X dev command line option. For example, PYTHONMALLOC=malloc disables pymalloc (forces the usage of malloc() of the C library).

I propose attached PR which removes the "m" flag from SOABI in Python 3.8 when pymalloc is enabled.

[vstinner]

I modified PyObject_MALLOC and other macros when I implemented the PEP-445 "Add new APIs to customize Python memory allocators":
https://www.python.org/dev/peps/pep-0445/

ABI tags are defined by the PEP-3149:
https://www.python.org/dev/peps/pep-3149/#proposal
"--with-pymalloc (flag: m)"

The PEP-393 already made "--with-wide-unicode (flag: u)" useless in Python 3.3.

[serhiy-storchaka]

If --with-pymalloc does not impact the API, why we need this option?

[vstinner]

pymalloc is enabled by default. --without-pymalloc allows to build Python
without pymalloc and use malloc by default.

[vstinner]

New changeset 6c44fde by Victor Stinner in branch 'master':
bpo-36707: Remove the "m" flag (pymalloc) from SOABI (GH-12931)
6c44fde

[ned-deily]

This change has an impact on downstream packagers and users of Python 3.8 because the SOABI flag values are also used to construct a number of file names and directories names in an Unix or macOS framework install besides the extension module (*.so) file names. Suggest comparing the directory/file names installed to an alternate prefix (./configure --prefix=/tmp/3x ; make install) in 3.7 versus master with this PR. Also suggest reading PEP-3149, which introduced the ABI-specific file names, and the various issues in the bug tracker that reference it. At the very least, these changes need to be documented in the What's New for 3.8. But the impact of the change should probably also be discussed with at least some of the large distributors. (The change already inadvertently broke the macOS installer build - fixing.)

[vstinner]

About the PEP, I asked Barry to review the change and he approved it.

About the doc, I didn't know that anyone rely on the exact filename. I documented all changes in a top-level section of the What's New in Python 3.8:
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

Sorry for thr annoyance. I didn't know that the macOS installer rely on these filenames.

[ned-deily]

No, you didn't document all the changed file names, See, for instance, /usr/bin/python3* for a start :)

[vstinner]

No, you didn't document all the changed file names, See, for instance, /usr/bin/python3* for a start :)

Oh, I see. I was thinking at something else. First I proposed to drop the "d" from the SOABI for debug build, but then I changed my mind. So yeah, I didn't change the SOABI for:
https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

And this change should be documented elsewhere.

[hroncok]

But the impact of the change should probably also be discussed with at least some of the large distributors.

Adapting the Fedora package. Will try to mass rebuild our packages to see what breaks.

[hroncok]

It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough. However, this change still is not documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html

[vstinner]

It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough. However, this change still is not documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html

I concur that it should be documented. It's on my TODO list. But I'm not sure of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m". Someone has to propose a PR to document that.

[hroncok]

I'm not sure either. So far most affected buildscripts are confused by:

/usr/include/python3.Xm -> /usr/include/python3.X
/usr/lib64/libpython3.Xm -> /usr/lib64/libpython3.X
extension.cpython-3Xm-arch-linux-gnu.so -> extension.cpython-3X-arch-linux-gnu.so

However this is pretty much Linux specific, so I wouldn't know how to explain this in an OS agnostic way.

[koobs]

For FreeBSD Python language ports, the change doesn't have a big negative impact for the Python language/interpreter ports themselves

We have the following block in lang/python3? ports:

.if ${PORT_OPTIONS:MPYMALLOC}
ABIFLAGS:= m${ABIFLAGS}
.endif

.if ${PORT_OPTIONS:MDEBUG}
ABIFLAGS:= d${ABIFLAGS}
.endif

We then use the variable to substitute the correct suffix into the pkg-plist items:

.if !empty(ABIFLAGS)
PLIST_FILES+= bin/python${PYTHON_VER}${ABIFLAGS} \
bin/python${PYTHON_VER}${ABIFLAGS}-config \
libdata/pkgconfig/python-${PYTHON_VER}${ABIFLAGS}.pc
<snip>
${INSTALL_DATA} ${WRKSRC}/Tools/gdb/libpython.py \
${STAGEDIR}${PREFIX}/lib/libpython${PYTHON_VER}${ABIFLAGS}.so.1.0-gdb.py

One question I have is, can/does SOABI flag removal affect any third party package/extension builds in any way, particularly affecting the names of files they produce?

We have a ton of python packages/extensions in the ports tree, so anything that affected their builds, particularly when it comes to setuptools --record output changes, would be a blocker for supporting 3.8

For example, we had to produce custom ports framework code to account for PEP-488 [1] and PEP-3147 [2] filename suffixes, for these filename differences, which was painful at the time:

# PEP-0488 (https://www.python.org/dev/peps/pep-0488/)
.if ${PYTHON_REL} < 3500
PYTHON_PYOEXTENSION= pyo
.else
PYTHON_PYOEXTENSION= opt-1.pyc
.endif

.if ${PYTHON_REL} >= 3200 && defined(PYTHON_FEATURE_PY3KPLIST)
# When Python version is 3.2+ we rewrite all the filenames
# of TMPPLIST that end with .py[co], so that they conform
# to PEP-3147 (see https://www.python.org/dev/peps/pep-3147/)
PYMAGICTAG= ${PYTHON_CMD} -c 'import sys; print(sys.implementation.cache_tag)'
_USES_stage+= 935:add-plist-python
add-plist-python:
@${AWK} '\
/\.py[co]$$/ && !($$0 ~ "/" pc "/") {id = match($$0, /\/[^\\/]+\.py[co]$$/); if (id != 0) {d = substr($$0, 1, RSTART - 1); dirs[d] = 1}; sub(/\.pyc$$/, "." mt "&"); sub(/\.pyo$$/, "." mt "." pyo); sub(/[^\\/]+\.py[co]$$/, pc "/&"); print; next} \
/^@dirrm / {d = substr($$0, 8); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
/^@dirrmtry / {d = substr($$0, 11); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
{print} \
' \
pc="__pycache_" mt="$$(${PYMAGICTAG})" pyo="opt-1.pyc" \
${TMPPLIST} > ${TMPPLIST}.pyc_tmp
@${MV} ${TMPPLIST}.pyc_tmp ${TMPPLIST}

[1] https://www.python.org/dev/peps/pep-0488/
[2] https://www.python.org/dev/peps/pep-3147/

[ned-deily]

One question I have is, can/does SOABI flag removal affect any third party package/extension builds in any way, particularly affecting the names of files they produce?

Yep, the default file names of C extension modules differ on most/all? Unix-y platforms (certainly vanilla Linux and macOS) because the SOABI is included in the file name.

$ /tmp/py38/bin/python3.7 -m pip install --no-binary :all: psutil
$ ls /tmp/py37/lib/python3.7/site-packages/psutil/*.so
/tmp/py37/lib/python3.7/site-packages/psutil/_psutil_linux.cpython-37m-i386-linux-gnu.so
/tmp/py37/lib/python3.7/site-packages/psutil/_psutil_posix.cpython-37m-i386-linux-gnu.so

$ /tmp/py38/bin/python3.8 -m pip install --no-binary :all: psutil
$ ls /tmp/py38/lib/python3.8/site-packages/psutil/*.so
/tmp/py38/lib/python3.8/site-packages/psutil/_psutil_linux.cpython-38-i386-linux-gnu.so
/tmp/py38/lib/python3.8/site-packages/psutil/_psutil_posix.cpython-38-i386-linux-gnu.so

[ned-deily]

Another user-visible difference:

$ ls /tmp/py37/bin/python*
/tmp/py37/bin/python3    /tmp/py37/bin/python3.7-config  /tmp/py37/bin/python3.7m-config
/tmp/py37/bin/python3.7  /tmp/py37/bin/python3.7m        /tmp/py37/bin/python3-config
$ ls /tmp/py38/bin/python*
/tmp/py38/bin/python3    /tmp/py38/bin/python3.8-config
/tmp/py38/bin/python3.8  /tmp/py38/bin/python3-config

Granted, these differences aren't the end of the world but:

"But I'm not sure of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m"."

Not to beat a dead horse but - we really should not make changes like this without first understanding what the impact will be. As it stands, this change potentially affects end-users, third-party distributors of Python, and package developers. And, for what reason? I don't wsee how this change provides any real value for anyone and it does cause people to have to change things, like creating conditional code for 3.7- and 3.8+. If we were starting from scratch today, sure - "m" is not needed. But just because we can change something doesn't mean we should.

[koobs]

Those script names would already have code to handle name changes based on existing abiflags/soabi string (as FreeBSD does), and would be trivial to modify for 'm' removal.

For third party stuff: we use setuptools --record for 'everything', so as long as filenames are 'correct' for the files that are actually installed, and downstreams are clear what the scope of the changes are, 'm' removal is not an issue when it come so packaging third party packages/extensions.

A new file extension format or substantially more complex magic would be, on the other hand.

A new minor is a good time to make these changes, and there's nothing really blocking about a new python38 port/package, and nothing consumes it yet in downstream OS's

[vstinner]

New changeset 7efc526 by Victor Stinner in branch 'master':
bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
7efc526

[miss-islington]

New changeset 3fde750 by Miss Islington (bot) in branch '3.8':
bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
3fde750

[vstinner]

Ned Deily:

My preference remains to not make the flags change at all

Well, "m" in ABI flags became useless around Python 3.4. I don't see the point of pretending that the "m" ABI is different and then provide two names for the same thing:

python3.8 and python3.8m
python3.8-config and python3.8m-config
man python3.8 and man python3.8m
etc.

I'm not sure why, but on Fedora /usr/bin/python3.7 and /usr/bin/python3.7m are two separated files... with the same content (ex: same MD5 sum). Well, it's just 17 KiB each :-) /usr/bin/python3.7-config is a symlink to python3.7m-config.

The change itself is painful for distributors like Red Hat (Fedora, RHEL), but things should be simpler once the change is done.

My long-term goal is to get a single ABI for everything: CPython, PyPy, RustPython, MicroPython, etc. :-) Maybe it's not possible, but at least I would like to move towards this goal!

Overall project:

• https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build
• https://pythoncapi.readthedocs.io/

[vstinner]

Thanks everybody for your help. It is now documented.