2.7: wrong user site directory on Linux

[srid]

BPO	8759
Nosy	@ronaldoussoren, @tarekziade, @merwok

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tarekziade'
closed_at = <Date 2010-05-21.01:58:20.381>
created_at = <Date 2010-05-18.21:57:43.065>
labels = ['type-bug', 'library']
title = '2.7: wrong user site directory on Linux'
updated_at = <Date 2010-08-27.01:21:46.150>
user = 'https://bugs.python.org/srid'

bugs.python.org fields:

activity = <Date 2010-08-27.01:21:46.150>
actor = 'eric.araujo'
assignee = 'tarek'
closed = True
closed_date = <Date 2010-05-21.01:58:20.381>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2010-05-18.21:57:43.065>
creator = 'srid'
dependencies = []
files = []
hgrepos = []
issue_num = 8759
keywords = []
message_count = 10.0
messages = ['106009', '106044', '106057', '106106', '106108', '106109', '106110', '106133', '106187', '115050']
nosy_count = 4.0
nosy_names = ['ronaldoussoren', 'tarek', 'eric.araujo', 'srid']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8759'
versions = ['Python 2.7']

[srid]

sridharr@triple:~$ /opt/ActivePython-2.7/bin/python -c "import sys; print sys.path; import cmdln"
['', '/opt/ActivePython-2.7/lib/python27.zip', '/opt/ActivePython-2.7/lib/python2.7', '/opt/ActivePython-2.7/lib/python2.7/plat-linux2', '/opt/ActivePython-2.7/lib/python2.7/lib-tk', '/opt/ActivePython-2.7/lib/python2.7/lib-old', '/opt/ActivePython-2.7/lib/python2.7/lib-dynload', '/home/sridharr/.local/lib/python/2.7/site-packages', '/opt/ActivePython-2.7/lib/python2.7/site-packages']
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named cmdln
sridharr@triple:~$ file /home/sridharr/.local/lib/python2.7/site-packages/cmdln.py 
/home/sridharr/.local/lib/python2.7/site-packages/cmdln.py: a python script text executable

On OSX, I don't even see a ~/.local directory!

python2.7 -c "import sys; print sys.path"
['', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages']

Works as expected on Windows, though.

[tarekziade]

Are you sure you didn't deactivate use site ? (by turning site.ENABLE_USER_SITE off)

Please check its value:

  >>> import site
  >>> site.ENABLE_USER_SITE 
  True

Pathes look fine on my Mac :

Python 2.7b2+ (trunk:81228, May 16 2010, 14:42:34) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/site-packages/pip-0.6.1-py2.7.egg', '/usr/local/lib/python2.7/site-packages/distribute-0.6.11dev-py2.7.egg', '/usr/local/lib/python27.zip', '/MacDev/svn.python.org/python-trunk/Lib', '/MacDev/svn.python.org/python-trunk/Lib/plat-darwin', '/MacDev/svn.python.org/python-trunk/Lib/plat-mac', '/MacDev/svn.python.org/python-trunk/Lib/plat-mac/lib-scriptpackages', '/MacDev/svn.python.org/python-trunk/Lib/lib-tk', '/MacDev/svn.python.org/python-trunk/Lib/lib-old', '/MacDev/svn.python.org/python-trunk/build/lib.macosx-10.4-i386-2.7', '/Users/tarek/.local/lib/python/2.7/site-packages', '/usr/local/lib/python2.7/site-packages']

If you still have the issue, you need to trace python load in site.py, where all the paths are added, by printing them out for example, and using the -v flag

[ronaldoussoren]

The --user directory for framework installs of python on OSX has changed from a subdirectory of ~/.local to a subdirectory ~/Library. As described in the NEWS file:

• Issue bpo-8084: PEP-370 now conforms to system conventions for framework
  builds on MacOS X. That is, "python setup.py install --user" will install
  into "~/Library/Python/2.7" instead of "~/.local".

In python 2.6 both ~/.local and ~/Library/Python were added to sys path and that's confusing. As ~/.local does not conform to the filesystem layout conventions on OSX the --user directory now always refers to ~/Library/Python.

For classic unix installs --user still uses the unix conventions. I'm not 100% sure that that is the right choice.

In other words: works as designed.

[srid]

@ronald: Ah, I see. So on Mac, Python 2.7 (with default site configuration) will not pickup packages installed in ~/.local, correct? It will, from now onwards, only consider ~/Library/Package as the user site directory? I ask because I just want to double check before modifying PyPM (which has ~/.local hardcoded in the client) to support this change.

@tarek: I have user site enabled; didn't change anything .. just the default settings:
$ /tmp/apy27/bin/python -m site
sys.path = [
'/home/sridharr/as/apy/trunk',
'/tmp/apy27/lib/python27.zip',
'/tmp/apy27/lib/python2.7',
'/tmp/apy27/lib/python2.7/plat-linux2',
'/tmp/apy27/lib/python2.7/lib-tk',
'/tmp/apy27/lib/python2.7/lib-old',
'/tmp/apy27/lib/python2.7/lib-dynload',
'/home/sridharr/.local/lib/python/2.7/site-packages',
'/tmp/apy27/lib/python2.7/site-packages',
'/tmp/apy27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info',
]
USER_BASE: '/home/sridharr/.local' (exists)
USER_SITE: '/home/sridharr/.local/lib/python/2.7/site-packages' (exists)
ENABLE_USER_SITE: True

...

That is strange user site location - "python/2.7/site-packages" really? Shouldn't it be "python2.7/site-packages"?

[srid]

On further analysis:

$ /tmp/apy27/bin/python -c "from sysconfig import get_path; print get_path('purelib', 'posix_user')"
/home/sridharr/.local/lib/python/2.7/site-packages
$

From sysconfig.py:

'posix_user': {
    'stdlib': '{userbase}/lib/python/{py_version_short}',
    'platstdlib': '{userbase}/lib/python/{py_version_short}',
    'purelib': '{userbase}/lib/python/{py_version_short}/site-packages',
    'platlib': '{userbase}/lib/python/{py_version_short}/site-packages',

Ah, there is the bug!

[tarekziade]

"python/2.7/site-packages" is a typo in sysconfig, it should be "python2.7/site-packages" you are right.

Fixing it..

[tarekziade]

Fixed in r81371, r81372

Thanks!

I let you close the issue once you are through with the rest of the talk.

[ronaldoussoren]

Shridar: that is correct a framework install will not look in ~/.local, but only in ~/Library/Python/2.7 (there is an install scheme in distutils that describes the exact layout).

Technically the name "Python" subdirectory is sysconfig.get_config_var("PYTHONFRAMEWORK"), to ensure that frameworks with different values for --with-framework-name will use different subdirectories.

[srid]

Thanks.

there is an install scheme in distutils that describes the exact layout

I figured that this is now exposed via the sysconfig module. There is also an old install scheme in distutils.command.install, but it seems to be outdated and is only applicable for 2.6.

My understanding is that, installers such as PyPM should use the scheme in distutils.command.install on 2.6 and the new sysconfig scheme for 2.7+.

Please feel free to close this issue (I can do it, but do not have the privileges to set the resolution field.)

[merwok]

Fixed in the distutils2 backport in 7f0148944e28 (not pushed yet due to SSH problem with Bitbucket).