Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

Closed
uranusjr mannequin opened this issue Aug 7, 2021 · 18 comments
Closed

sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

uranusjr mannequin opened this issue Aug 7, 2021 · 18 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes release-blocker stdlib Python modules in the Lib dir

Comments

@uranusjr
Copy link
Mannequin

uranusjr mannequin commented Aug 7, 2021

BPO 44860
Nosy @vstinner, @merwok, @encukou, @ambv, @dstufft, @uranusjr, @hroncok, @frenzymadness, @pablogsal, @miss-islington
PRs
  • bpo-44860: Make sysconfig posix_user not depend on platlibdir #27655
  • [3.10] bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) #28197
  • bpo-44860: Update test_sysconfig for posix_user platlib #28235
  • [3.10] bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) #28251
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-09-07.11:46:12.337>
    created_at = <Date 2021-08-07.14:29:18.323>
    labels = ['release-blocker', 'library', '3.9', '3.10', '3.11']
    title = "sysconfig's posix_user scheme has different platlib value to distutils's unix_user"
    updated_at = <Date 2021-10-04.19:18:43.703>
    user = 'https://github.com/uranusjr'

    bugs.python.org fields:

    activity = <Date 2021-10-04.19:18:43.703>
    actor = 'pablogsal'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-09-07.11:46:12.337>
    closer = 'pablogsal'
    components = ['Distutils']
    creation = <Date 2021-08-07.14:29:18.323>
    creator = 'uranusjr'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 44860
    keywords = ['patch']
    message_count = 18.0
    messages = ['399186', '399315', '399316', '399318', '399326', '401217', '401218', '401221', '401241', '401242', '401250', '401264', '401266', '401373', '401448', '401452', '401455', '403172']
    nosy_count = 10.0
    nosy_names = ['vstinner', 'eric.araujo', 'petr.viktorin', 'lukasz.langa', 'dstufft', 'uranusjr', 'hroncok', 'frenzy', 'pablogsal', 'miss-islington']
    pr_nums = ['27655', '28197', '28235', '28251']
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue44860'
    versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    @uranusjr
    Copy link
    Mannequin Author

    uranusjr mannequin commented Aug 7, 2021

    On POSIX, the user scheme has a different 'platlib' location between distutils and sysconfig, dispite the comment claiming they should be the same.

    This can be reproduced on Fedora 34's stock Python 3.9:

        $ docker run -it --rm -h=p fedora:34 bash
        ...
        [root@p /]# yum install python3 -y
        ...
        [root@p /]# type python3
        python3 is hashed (/usr/bin/python3)
        [root@p /]# python3 -V
        Python 3.9.6
        [root@p /]# python3.9 -q
        >>> from distutils.command.install import install
        >>> from distutils.dist import Distribution
        >>> c = install(Distribution())
        >>> c.user = True
        >>> c.finalize_options()
        >>> c.install_platlib
        '/root/.local/lib/python3.9/site-packages'
        >>> import sysconfig
        >>> sysconfig.get_path('platlib', 'posix_user')
        '/root/.local/lib64/python3.9/site-packages'

    This issue was introduced by the sys.platlibdir value, and its usage in distutils and sysconfig. sysconfig sets posix_user's lib paths like this:

    'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
    'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages',
    

    cpython/Lib/sysconfig.py

    Lines 100 to 108 in a40675c

    'posix_user': {
    'stdlib': '{userbase}/{platlibdir}/python{py_version_short}',
    'platstdlib': '{userbase}/{platlibdir}/python{py_version_short}',
    'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
    'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages',
    'include': '{userbase}/include/python{py_version_short}',
    'scripts': '{userbase}/bin',
    'data': '{userbase}',
    },

    But distutils naively sets both to the same value that does not account for platlibdir:

    'purelib': '$usersite',
    'platlib': '$usersite',
    

    # The following part of INSTALL_SCHEMES has a different definition
    # than the one in sysconfig, but because both depend on the site module,
    # the outcomes should be the same.
    if HAS_USER_SITE:
    INSTALL_SCHEMES['nt_user'] = {
    'purelib': '$usersite',
    'platlib': '$usersite',
    'headers': '$userbase/Python$py_version_nodot/Include/$dist_name',
    'scripts': '$userbase/Python$py_version_nodot/Scripts',
    'data' : '$userbase',
    }
    INSTALL_SCHEMES['unix_user'] = {
    'purelib': '$usersite',
    'platlib': '$usersite',
    'headers':
    '$userbase/include/python$py_version_short$abiflags/$dist_name',
    'scripts': '$userbase/bin',
    'data' : '$userbase',
    }

    causing the mismatch, dispite the comment above clearly indicating the values are supposed to be the same.

    This was introduced in bpo-1294959 which changed the platlib template to depend on sys.platlibdir, so a mismatch happens when the value of sys.platlibdir is not 'lib'.

    (Adding frenzy and vstinner to the nosy list since you introduced the comment in distutils and the sys.platlibdir change, respectively.)

    @uranusjr uranusjr mannequin added 3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes stdlib Python modules in the Lib dir labels Aug 7, 2021
    @vstinner
    Copy link
    Member

    sys.platlibdir was introduced to install libraries in /usr/lib64 rather than /usr/lib. I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

    Does the site module add $HOME/.local/lib64 to sys.path if it exists?

    @uranusjr
    Copy link
    Mannequin Author

    uranusjr mannequin commented Aug 10, 2021

    I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

    This was also briefly discussed in bpo-1294959, but did not go through since “changing posix_user should have no impact on end users”.

    Does the site module add $HOME/.local/lib64 to sys.path if it exists?

    It does not, only lib is checked right now.

    cpython/Lib/site.py

    Lines 288 to 298 in c7ea1e3

    def _get_path(userbase):
    version = sys.version_info
    if os.name == 'nt':
    ver_nodot = sys.winver.replace('.', '')
    return f'{userbase}\\Python{ver_nodot}\\site-packages'
    if sys.platform == 'darwin' and sys._framework:
    return f'{userbase}/lib/python/site-packages'
    return f'{userbase}/lib/python{version[0]}.{version[1]}/site-packages'

    ----

    There are two possible solutions from what I can tell. We could just make posix_user match posix_prefix and always respect sys.platlibdir. This could be confusing to existing Python 3.9 users however since many of them already pip-installed things into ~/.local/lib and this would make their user-site packages split in two locations. The other would be to restore the pre-3.9 behaviour in sysconfig to use lib instead of depending on sys.platlibdir. I don’t know who uses sysconfig right now and can’t say what would break, but for pip this would be less disruptive since it currently installs things into ~/.local/ib (provided by distutils).

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Aug 10, 2021

    Installing to ~/.local/lib works, installing to ~/.local/lib64 breaks things, as it is not on sys.path. I agree that restoring the pre-3.9 behavior in sysconfig to use lib instead of depending on sys.platlibdir is a better fix, at least for 3.9 and 3.10. We can redesign things in 3.11 (but I wouldn't).

    @uranusjr
    Copy link
    Mannequin Author

    uranusjr mannequin commented Aug 10, 2021

    I’ve updated the linked PR to change sysconfig instead to not use sys.platlibdir when generating the posix_user scheme. This means posix_user would behave the same in 3.9+ as 3.8 and prior.

    @encukou
    Copy link
    Member

    encukou commented Sep 7, 2021

    Marking as *potential* release blocker for 3.10.

    Pablo, without this change the newest pip (with [PR 10358]) will not work on Python 3.10 built --with-platlibdir.
    This configure option was added in 3.9 for distros that separate /usr/lib and /usr/lib64, which used downstream patches before.
    Things will be broken if:

    • this bug is not fixed in 3.10
    • AND pip switches to _USE_SYSCONFIG in 3.10 (with merged [PR 10358])
    • AND distros don't patch again.

    So, if this isn't merged in 3.10, either pip or downstreams will need to implement a workaround. (If pip doesn't hold off, downstreams that build --with-platlibdir will likely carry this exact patch.)
    On the other hand, we're very late in the release cycle.

    Note that there's a similar bug in bpo-45035

    [PR 10358]: pypa/pip#10358

    @pablogsal
    Copy link
    Member

    IIUC we need to backport PR27655 to 3.10 no? Or do we need something else?

    @encukou
    Copy link
    Member

    encukou commented Sep 7, 2021

    Possibly together with PR28011.

    @pablogsal
    Copy link
    Member

    New changeset 608a629 by Miss Islington (bot) in branch '3.10':
    bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) (GH-28197)
    608a629

    @pablogsal
    Copy link
    Member

    Petr, is something left to do for this release blocker? I am planning to start the release if everything is OK

    @encukou
    Copy link
    Member

    encukou commented Sep 7, 2021

    I believe everything is in order now.

    @vstinner
    Copy link
    Member

    vstinner commented Sep 7, 2021

    @vstinner
    Copy link
    Member

    vstinner commented Sep 7, 2021

    Let's continue the discussion for the unix_home scheme in bpo-45035.

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented Sep 8, 2021

    There seem to be a regression in test_user_similar: https://bugs.python.org/issue45136

    @vstinner
    Copy link
    Member

    vstinner commented Sep 9, 2021

    New changeset 49acac0 by Victor Stinner in branch 'main':
    bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235)
    49acac0

    @vstinner
    Copy link
    Member

    vstinner commented Sep 9, 2021

    New changeset 11103eb by Miss Islington (bot) in branch '3.10':
    bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
    11103eb

    @vstinner
    Copy link
    Member

    vstinner commented Sep 9, 2021

    I marked bpo-45136 as a duplicate of this issue.

    @pablogsal
    Copy link
    Member

    New changeset 01be518 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
    bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
    01be518

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 bug and security fixes release-blocker stdlib Python modules in the Lib dir
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants