sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

uranusjr · 2021-08-07T14:29:18Z

BPO	44860
Nosy	@vstinner, @merwok, @encukou, @ambv, @dstufft, @uranusjr, @hroncok, @frenzymadness, @pablogsal, @miss-islington
PRs	bpo-44860: Make sysconfig posix_user not depend on platlibdir #27655 [3.10] bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) #28197 bpo-44860: Update test_sysconfig for posix_user platlib #28235 [3.10] bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) #28251

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-07.11:46:12.337>
created_at = <Date 2021-08-07.14:29:18.323>
labels = ['release-blocker', 'library', '3.9', '3.10', '3.11']
title = "sysconfig's posix_user scheme has different platlib value to distutils's unix_user"
updated_at = <Date 2021-10-04.19:18:43.703>
user = 'https://github.com/uranusjr'

bugs.python.org fields:

activity = <Date 2021-10-04.19:18:43.703>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-07.11:46:12.337>
closer = 'pablogsal'
components = ['Distutils']
creation = <Date 2021-08-07.14:29:18.323>
creator = 'uranusjr'
dependencies = []
files = []
hgrepos = []
issue_num = 44860
keywords = ['patch']
message_count = 18.0
messages = ['399186', '399315', '399316', '399318', '399326', '401217', '401218', '401221', '401241', '401242', '401250', '401264', '401266', '401373', '401448', '401452', '401455', '403172']
nosy_count = 10.0
nosy_names = ['vstinner', 'eric.araujo', 'petr.viktorin', 'lukasz.langa', 'dstufft', 'uranusjr', 'hroncok', 'frenzy', 'pablogsal', 'miss-islington']
pr_nums = ['27655', '28197', '28235', '28251']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44860'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

uranusjr · 2021-08-07T14:29:18Z

On POSIX, the user scheme has a different 'platlib' location between distutils and sysconfig, dispite the comment claiming they should be the same.

This can be reproduced on Fedora 34's stock Python 3.9:

    $ docker run -it --rm -h=p fedora:34 bash
    ...
    [root@p /]# yum install python3 -y
    ...
    [root@p /]# type python3
    python3 is hashed (/usr/bin/python3)
    [root@p /]# python3 -V
    Python 3.9.6
    [root@p /]# python3.9 -q
    >>> from distutils.command.install import install
    >>> from distutils.dist import Distribution
    >>> c = install(Distribution())
    >>> c.user = True
    >>> c.finalize_options()
    >>> c.install_platlib
    '/root/.local/lib/python3.9/site-packages'
    >>> import sysconfig
    >>> sysconfig.get_path('platlib', 'posix_user')
    '/root/.local/lib64/python3.9/site-packages'

This issue was introduced by the sys.platlibdir value, and its usage in distutils and sysconfig. sysconfig sets posix_user's lib paths like this:

'purelib': '{userbase}/lib/python{py_version_short}/site-packages',
'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages',

cpython/Lib/sysconfig.py

Lines 100 to 108 in a40675c

    
           'posix_user': { 
        
               'stdlib': '{userbase}/{platlibdir}/python{py_version_short}', 
        
               'platstdlib': '{userbase}/{platlibdir}/python{py_version_short}', 
        
               'purelib': '{userbase}/lib/python{py_version_short}/site-packages', 
        
               'platlib': '{userbase}/{platlibdir}/python{py_version_short}/site-packages', 
        
               'include': '{userbase}/include/python{py_version_short}', 
        
               'scripts': '{userbase}/bin', 
        
               'data': '{userbase}', 
        
               },

But distutils naively sets both to the same value that does not account for platlibdir:

'purelib': '$usersite',
'platlib': '$usersite',

cpython/Lib/distutils/command/install.py

Lines 68 to 87 in a40675c

    
           # The following part of INSTALL_SCHEMES has a different definition 
        
           # than the one in sysconfig, but because both depend on the site module, 
        
           # the outcomes should be the same. 
        
           if HAS_USER_SITE: 
        
               INSTALL_SCHEMES['nt_user'] = { 
        
                   'purelib': '$usersite', 
        
                   'platlib': '$usersite', 
        
                   'headers': '$userbase/Python$py_version_nodot/Include/$dist_name', 
        
                   'scripts': '$userbase/Python$py_version_nodot/Scripts', 
        
                   'data'   : '$userbase', 
        
                   } 
        
               INSTALL_SCHEMES['unix_user'] = { 
        
                   'purelib': '$usersite', 
        
                   'platlib': '$usersite', 
        
                   'headers': 
        
                       '$userbase/include/python$py_version_short$abiflags/$dist_name', 
        
                   'scripts': '$userbase/bin', 
        
                   'data'   : '$userbase', 
        
                   }

causing the mismatch, dispite the comment above clearly indicating the values are supposed to be the same.

This was introduced in bpo-1294959 which changed the platlib template to depend on sys.platlibdir, so a mismatch happens when the value of sys.platlibdir is not 'lib'.

(Adding frenzy and vstinner to the nosy list since you introduced the comment in distutils and the sys.platlibdir change, respectively.)

vstinner · 2021-08-10T11:11:01Z

sys.platlibdir was introduced to install libraries in /usr/lib64 rather than /usr/lib. I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

Does the site module add $HOME/.local/lib64 to sys.path if it exists?

uranusjr · 2021-08-10T12:10:15Z

I'm not sure if it should be used to install libraries in $HOME/.local/lib64 rather than $HOME/.local/lib. Previously, Fedora already used $HOME/.local/lib and $HOME/.local/lib64 is not in the sys.path.

This was also briefly discussed in bpo-1294959, but did not go through since “changing posix_user should have no impact on end users”.

Does the site module add $HOME/.local/lib64 to sys.path if it exists?

It does not, only lib is checked right now.

cpython/Lib/site.py

Lines 288 to 298 in c7ea1e3

    
           def _get_path(userbase): 
        
               version = sys.version_info 
        
               if os.name == 'nt': 
        
                   ver_nodot = sys.winver.replace('.', '') 
        
                   return f'{userbase}\\Python{ver_nodot}\\site-packages' 
        
               if sys.platform == 'darwin' and sys._framework: 
        
                   return f'{userbase}/lib/python/site-packages' 
        
               return f'{userbase}/lib/python{version[0]}.{version[1]}/site-packages'

----

There are two possible solutions from what I can tell. We could just make posix_user match posix_prefix and always respect sys.platlibdir. This could be confusing to existing Python 3.9 users however since many of them already pip-installed things into ~/.local/lib and this would make their user-site packages split in two locations. The other would be to restore the pre-3.9 behaviour in sysconfig to use lib instead of depending on sys.platlibdir. I don’t know who uses sysconfig right now and can’t say what would break, but for pip this would be less disruptive since it currently installs things into ~/.local/ib (provided by distutils).

hroncok · 2021-08-10T12:47:47Z

Installing to ~/.local/lib works, installing to ~/.local/lib64 breaks things, as it is not on sys.path. I agree that restoring the pre-3.9 behavior in sysconfig to use lib instead of depending on sys.platlibdir is a better fix, at least for 3.9 and 3.10. We can redesign things in 3.11 (but I wouldn't).

uranusjr · 2021-08-10T13:51:04Z

I’ve updated the linked PR to change sysconfig instead to not use sys.platlibdir when generating the posix_user scheme. This means posix_user would behave the same in 3.9+ as 3.8 and prior.

encukou · 2021-09-07T09:36:50Z

Marking as *potential* release blocker for 3.10.

Pablo, without this change the newest pip (with [PR 10358]) will not work on Python 3.10 built --with-platlibdir.
This configure option was added in 3.9 for distros that separate /usr/lib and /usr/lib64, which used downstream patches before.
Things will be broken if:

this bug is not fixed in 3.10
AND pip switches to _USE_SYSCONFIG in 3.10 (with merged [PR 10358])
AND distros don't patch again.

So, if this isn't merged in 3.10, either pip or downstreams will need to implement a workaround. (If pip doesn't hold off, downstreams that build --with-platlibdir will likely carry this exact patch.)
On the other hand, we're very late in the release cycle.

Note that there's a similar bug in bpo-45035

[PR 10358]: pypa/pip#10358

pablogsal · 2021-09-07T10:03:54Z

IIUC we need to backport PR27655 to 3.10 no? Or do we need something else?

encukou · 2021-09-07T10:19:26Z

Possibly together with PR28011.

pablogsal · 2021-09-07T11:20:36Z

New changeset 608a629 by Miss Islington (bot) in branch '3.10':
bpo-44860: Make sysconfig posix_user not depend on platlibdir (GH-27655) (GH-28197)
608a629

pablogsal · 2021-09-07T11:21:46Z

Petr, is something left to do for this release blocker? I am planning to start the release if everything is OK

encukou · 2021-09-07T11:50:01Z

I believe everything is in order now.

vstinner · 2021-09-07T13:20:04Z

See also the email on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/5UU6V2B3KBS4Z7OG5T7D6YQZASFNSBJM/

vstinner · 2021-09-07T13:23:15Z

Let's continue the discussion for the unix_home scheme in bpo-45035.

hroncok · 2021-09-08T10:47:32Z

There seem to be a regression in test_user_similar: https://bugs.python.org/issue45136

vstinner · 2021-09-09T09:02:58Z

New changeset 49acac0 by Victor Stinner in branch 'main':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235)
49acac0

vstinner · 2021-09-09T09:36:01Z

New changeset 11103eb by Miss Islington (bot) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
11103eb

vstinner · 2021-09-09T09:54:22Z

I marked bpo-45136 as a duplicate of this issue.

pablogsal · 2021-10-04T19:18:44Z

New changeset 01be518 by Pablo Galindo (Miss Islington (bot)) in branch '3.10':
bpo-44860: Update test_sysconfig for posix_user platlib (GH-28235) (GH-28251)
01be518

uranusjr mannequin added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir labels Aug 7, 2021

encukou added the release-blocker label Sep 7, 2021

pablogsal closed this as completed Sep 7, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

uranusjr mannequin commented Aug 7, 2021

uranusjr mannequin commented Aug 7, 2021

vstinner commented Aug 10, 2021

uranusjr mannequin commented Aug 10, 2021

hroncok mannequin commented Aug 10, 2021

uranusjr mannequin commented Aug 10, 2021

encukou commented Sep 7, 2021

pablogsal commented Sep 7, 2021

encukou commented Sep 7, 2021

pablogsal commented Sep 7, 2021

pablogsal commented Sep 7, 2021

encukou commented Sep 7, 2021

vstinner commented Sep 7, 2021

vstinner commented Sep 7, 2021

hroncok mannequin commented Sep 8, 2021

vstinner commented Sep 9, 2021

vstinner commented Sep 9, 2021

vstinner commented Sep 9, 2021

pablogsal commented Oct 4, 2021

sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

sysconfig's posix_user scheme has different platlib value to distutils's unix_user #89023

Comments

uranusjr mannequin commented Aug 7, 2021

uranusjr mannequin commented Aug 7, 2021

vstinner commented Aug 10, 2021

uranusjr mannequin commented Aug 10, 2021

hroncok mannequin commented Aug 10, 2021

uranusjr mannequin commented Aug 10, 2021

encukou commented Sep 7, 2021

pablogsal commented Sep 7, 2021

encukou commented Sep 7, 2021

pablogsal commented Sep 7, 2021

pablogsal commented Sep 7, 2021

encukou commented Sep 7, 2021

vstinner commented Sep 7, 2021

vstinner commented Sep 7, 2021

hroncok mannequin commented Sep 8, 2021

vstinner commented Sep 9, 2021

vstinner commented Sep 9, 2021

vstinner commented Sep 9, 2021

pablogsal commented Oct 4, 2021