Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The "m" ABI flag of SOABI for pymalloc is no longer needed #80888

Closed
vstinner opened this issue Apr 24, 2019 · 21 comments
Closed

The "m" ABI flag of SOABI for pymalloc is no longer needed #80888

vstinner opened this issue Apr 24, 2019 · 21 comments
Assignees
Labels
3.8 build The build process and cross-build deferred-blocker

Comments

@vstinner
Copy link
Member

vstinner commented Apr 24, 2019

BPO 36707
Nosy @vstinner, @ned-deily, @serhiy-storchaka, @koobs, @hroncok, @miss-islington
PRs
  • bpo-36707: Remove "m" flag from SOABI #12931
  • bpo-36707: Document "m" removed from sys.abiflags #14090
  • [3.8] bpo-36707: Document "m" removal from sys.abiflags (GH-14090) #14097
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/vstinner'
    closed_at = <Date 2019-06-15.09:36:07.714>
    created_at = <Date 2019-04-24.00:27:25.531>
    labels = ['deferred-blocker', 'build', '3.8']
    title = 'The "m" ABI flag of SOABI for pymalloc is no longer needed'
    updated_at = <Date 2019-06-15.09:39:47.645>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2019-06-15.09:39:47.645>
    actor = 'vstinner'
    assignee = 'vstinner'
    closed = True
    closed_date = <Date 2019-06-15.09:36:07.714>
    closer = 'vstinner'
    components = ['Build']
    creation = <Date 2019-04-24.00:27:25.531>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36707
    keywords = ['patch']
    message_count = 21.0
    messages = ['340748', '340749', '340757', '340766', '340773', '341097', '341099', '341100', '341121', '341835', '344478', '344483', '344484', '344521', '344524', '344526', '344528', '345650', '345651', '345666', '345667']
    nosy_count = 6.0
    nosy_names = ['vstinner', 'ned.deily', 'serhiy.storchaka', 'koobs', 'hroncok', 'miss-islington']
    pr_nums = ['12931', '14090', '14097']
    priority = 'deferred blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue36707'
    versions = ['Python 3.8']

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 24, 2019

    In Python 2.7, the WITH_PYMALLOC define (defined by ./configure --with-pymalloc which is the default) modified the Python API. For example, PyObject_MALLOC() is a macro replaced with PyObject_Malloc() with pymalloc but replaced with PyMem_MALLOC() without pymalloc.

    In Python 3.8, it's no longer the case, PyObject_MALLOC is no longer modified by WITH_PYMALLOC, it's always an alias to PyObject_Malloc() for backward compatibility.

    #define PyObject_MALLOC   PyObject_Malloc

    More generally, WITH_PYMALLOC has no impact on the Python headers nor on the ABI in any way. Memory allocators can be switched at runtime using PYTHONMALLOC environment variable and -X dev command line option. For example, PYTHONMALLOC=malloc disables pymalloc (forces the usage of malloc() of the C library).

    I propose attached PR which removes the "m" flag from SOABI in Python 3.8 when pymalloc is enabled.

    @vstinner vstinner added the 3.8 label Apr 24, 2019
    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 24, 2019

    I modified PyObject_MALLOC and other macros when I implemented the PEP-445 "Add new APIs to customize Python memory allocators":
    https://www.python.org/dev/peps/pep-0445/

    ABI tags are defined by the PEP-3149:
    https://www.python.org/dev/peps/pep-3149/#proposal
    "--with-pymalloc (flag: m)"

    The PEP-393 already made "--with-wide-unicode (flag: u)" useless in Python 3.3.

    @serhiy-storchaka
    Copy link
    Member

    serhiy-storchaka commented Apr 24, 2019

    If --with-pymalloc does not impact the API, why we need this option?

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 24, 2019

    pymalloc is enabled by default. --without-pymalloc allows to build Python
    without pymalloc and use malloc by default.

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 24, 2019

    New changeset 6c44fde by Victor Stinner in branch 'master':
    bpo-36707: Remove the "m" flag (pymalloc) from SOABI (GH-12931)
    6c44fde

    @vstinner vstinner added the build The build process and cross-build label Apr 24, 2019
    @ned-deily
    Copy link
    Member

    ned-deily commented Apr 29, 2019

    This change has an impact on downstream packagers and users of Python 3.8 because the SOABI flag values are also used to construct a number of file names and directories names in an Unix or macOS framework install besides the extension module (*.so) file names. Suggest comparing the directory/file names installed to an alternate prefix (./configure --prefix=/tmp/3x ; make install) in 3.7 versus master with this PR. Also suggest reading PEP-3149, which introduced the ABI-specific file names, and the various issues in the bug tracker that reference it. At the very least, these changes need to be documented in the What's New for 3.8. But the impact of the change should probably also be discussed with at least some of the large distributors. (The change already inadvertently broke the macOS installer build - fixing.)

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 29, 2019

    About the PEP, I asked Barry to review the change and he approved it.

    About the doc, I didn't know that anyone rely on the exact filename. I documented all changes in a top-level section of the What's New in Python 3.8:
    https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

    Sorry for thr annoyance. I didn't know that the macOS installer rely on these filenames.

    @ned-deily
    Copy link
    Member

    ned-deily commented Apr 29, 2019

    No, you didn't document all the changed file names, See, for instance, /usr/bin/python3* for a start :)

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 29, 2019

    No, you didn't document all the changed file names, See, for instance, /usr/bin/python3* for a start :)

    Oh, I see. I was thinking at something else. First I proposed to drop the "d" from the SOABI for debug build, but then I changed my mind. So yeah, I didn't change the SOABI for:
    https://docs.python.org/dev/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

    And this change should be documented elsewhere.

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented May 7, 2019

    But the impact of the change should probably also be discussed with at least some of the large distributors.

    Adapting the Fedora package. Will try to mass rebuild our packages to see what breaks.

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Jun 3, 2019

    It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough. However, this change still is not documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 3, 2019

    It's usually just custom ugly-hardcoded autotools/cmake. Nothing severe enough. However, this change still is not documented anywhere in https://docs.python.org/dev/whatsnew/3.8.html

    I concur that it should be documented. It's on my TODO list. But I'm not sure of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m". Someone has to propose a PR to document that.

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Jun 3, 2019

    I'm not sure either. So far most affected buildscripts are confused by:

    /usr/include/python3.Xm -> /usr/include/python3.X
    /usr/lib64/libpython3.Xm -> /usr/lib64/libpython3.X
    extension.cpython-3Xm-arch-linux-gnu.so -> extension.cpython-3X-arch-linux-gnu.so

    However this is pretty much Linux specific, so I wouldn't know how to explain this in an OS agnostic way.

    @koobs
    Copy link

    koobs commented Jun 4, 2019

    For FreeBSD Python language ports, the change doesn't have a big negative impact for the Python language/interpreter ports themselves

    We have the following block in lang/python3? ports:

    .if ${PORT_OPTIONS:MPYMALLOC}
    ABIFLAGS:= m${ABIFLAGS}
    .endif

    .if ${PORT_OPTIONS:MDEBUG}
    ABIFLAGS:= d${ABIFLAGS}
    .endif

    We then use the variable to substitute the correct suffix into the pkg-plist items:

    .if !empty(ABIFLAGS)
    PLIST_FILES+= bin/python${PYTHON_VER}${ABIFLAGS} \
    bin/python${PYTHON_VER}${ABIFLAGS}-config \
    libdata/pkgconfig/python-${PYTHON_VER}${ABIFLAGS}.pc
    <snip>
    ${INSTALL_DATA} ${WRKSRC}/Tools/gdb/libpython.py \
    ${STAGEDIR}${PREFIX}/lib/libpython${PYTHON_VER}${ABIFLAGS}.so.1.0-gdb.py

    One question I have is, can/does SOABI flag removal affect any third party package/extension builds in any way, particularly affecting the names of files they produce?

    We have a ton of python packages/extensions in the ports tree, so anything that affected their builds, particularly when it comes to setuptools --record output changes, would be a blocker for supporting 3.8

    For example, we had to produce custom ports framework code to account for PEP-488 [1] and PEP-3147 [2] filename suffixes, for these filename differences, which was painful at the time:

    # PEP-0488 (https://www.python.org/dev/peps/pep-0488/)
    .if ${PYTHON_REL} < 3500
    PYTHON_PYOEXTENSION= pyo
    .else
    PYTHON_PYOEXTENSION= opt-1.pyc
    .endif

    .if ${PYTHON_REL} >= 3200 && defined(PYTHON_FEATURE_PY3KPLIST)
    # When Python version is 3.2+ we rewrite all the filenames
    # of TMPPLIST that end with .py[co], so that they conform
    # to PEP-3147 (see https://www.python.org/dev/peps/pep-3147/)
    PYMAGICTAG= ${PYTHON_CMD} -c 'import sys; print(sys.implementation.cache_tag)'
    _USES_stage+= 935:add-plist-python
    add-plist-python:
    @${AWK} '\
    /\.py[co]$$/ && !($$0 ~ "/" pc "/") {id = match($$0, /\/[^\\/]+\.py[co]$$/); if (id != 0) {d = substr($$0, 1, RSTART - 1); dirs[d] = 1}; sub(/\.pyc$$/, "." mt "&"); sub(/\.pyo$$/, "." mt "." pyo); sub(/[^\\/]+\.py[co]$$/, pc "/&"); print; next} \
    /^@dirrm / {d = substr($$0, 8); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
    /^@dirrmtry / {d = substr($$0, 11); if (d in dirs) {print $$0 "/" pc}; print $$0; next} \
    {print} \
    ' \
    pc="__pycache
    _" mt="$$(${PYMAGICTAG})" pyo="opt-1.pyc" \
    ${TMPPLIST} &gt; ${TMPPLIST}.pyc_tmp
    @${MV} ${TMPPLIST}.pyc_tmp ${TMPPLIST}

    [1] https://www.python.org/dev/peps/pep-0488/
    [2] https://www.python.org/dev/peps/pep-3147/

    @ned-deily
    Copy link
    Member

    ned-deily commented Jun 4, 2019

    One question I have is, can/does SOABI flag removal affect any third party package/extension builds in any way, particularly affecting the names of files they produce?

    Yep, the default file names of C extension modules differ on most/all? Unix-y platforms (certainly vanilla Linux and macOS) because the SOABI is included in the file name.

    $ /tmp/py38/bin/python3.7 -m pip install --no-binary :all: psutil
    $ ls /tmp/py37/lib/python3.7/site-packages/psutil/*.so
    /tmp/py37/lib/python3.7/site-packages/psutil/_psutil_linux.cpython-37m-i386-linux-gnu.so
    /tmp/py37/lib/python3.7/site-packages/psutil/_psutil_posix.cpython-37m-i386-linux-gnu.so
    
    $ /tmp/py38/bin/python3.8 -m pip install --no-binary :all: psutil
    $ ls /tmp/py38/lib/python3.8/site-packages/psutil/*.so
    /tmp/py38/lib/python3.8/site-packages/psutil/_psutil_linux.cpython-38-i386-linux-gnu.so
    /tmp/py38/lib/python3.8/site-packages/psutil/_psutil_posix.cpython-38-i386-linux-gnu.so

    @ned-deily
    Copy link
    Member

    ned-deily commented Jun 4, 2019

    Another user-visible difference:

    $ ls /tmp/py37/bin/python*
    /tmp/py37/bin/python3    /tmp/py37/bin/python3.7-config  /tmp/py37/bin/python3.7m-config
    /tmp/py37/bin/python3.7  /tmp/py37/bin/python3.7m        /tmp/py37/bin/python3-config
    $ ls /tmp/py38/bin/python*
    /tmp/py38/bin/python3    /tmp/py38/bin/python3.8-config
    /tmp/py38/bin/python3.8  /tmp/py38/bin/python3-config

    Granted, these differences aren't the end of the world but:

    "But I'm not sure of what are the things impacting by ABIFLAGS="" rather than ABIFLAGS="m"."

    Not to beat a dead horse but - we really should not make changes like this without first understanding what the impact will be. As it stands, this change potentially affects end-users, third-party distributors of Python, and package developers. And, for what reason? I don't wsee how this change provides any real value for anyone and it does cause people to have to change things, like creating conditional code for 3.7- and 3.8+. If we were starting from scratch today, sure - "m" is not needed. But just because we can change something doesn't mean we should.

    @koobs
    Copy link

    koobs commented Jun 4, 2019

    Those script names would already have code to handle name changes based on existing abiflags/soabi string (as FreeBSD does), and would be trivial to modify for 'm' removal.

    For third party stuff: we use setuptools --record for 'everything', so as long as filenames are 'correct' for the files that are actually installed, and downstreams are clear what the scope of the changes are, 'm' removal is not an issue when it come so packaging third party packages/extensions.

    A new file extension format or substantially more complex magic would be, on the other hand.

    A new minor is a good time to make these changes, and there's nothing really blocking about a new python38 port/package, and nothing consumes it yet in downstream OS's

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 15, 2019

    New changeset 7efc526 by Victor Stinner in branch 'master':
    bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
    7efc526

    @miss-islington
    Copy link
    Contributor

    miss-islington commented Jun 15, 2019

    New changeset 3fde750 by Miss Islington (bot) in branch '3.8':
    bpo-36707: Document "m" removal from sys.abiflags (GH-14090)
    3fde750

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 15, 2019

    Ned Deily:

    My preference remains to not make the flags change at all

    Well, "m" in ABI flags became useless around Python 3.4. I don't see the point of pretending that the "m" ABI is different and then provide two names for the same thing:

    python3.8 and python3.8m
    python3.8-config and python3.8m-config
    man python3.8 and man python3.8m
    etc.
    

    I'm not sure why, but on Fedora /usr/bin/python3.7 and /usr/bin/python3.7m are two separated files... with the same content (ex: same MD5 sum). Well, it's just 17 KiB each :-) /usr/bin/python3.7-config is a symlink to python3.7m-config.

    The change itself is painful for distributors like Red Hat (Fedora, RHEL), but things should be simpler once the change is done.

    My long-term goal is to get a single ABI for everything: CPython, PyPy, RustPython, MicroPython, etc. :-) Maybe it's not possible, but at least I would like to move towards this goal!

    Overall project:

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 15, 2019

    Thanks everybody for your help. It is now documented.

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 build The build process and cross-build deferred-blocker
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants