system wide site-packages dir not used on Mac OS X

[kapet]

BPO	4865
Nosy	@loewis, @ronaldoussoren, @ned-deily
Files	apple.diff: Diff stock Python 2.5.1 against Apple Python

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2009-03-30.23:16:33.941>
created_at = <Date 2009-01-07.12:51:01.510>
labels = ['OS-mac', 'type-bug', 'library']
title = 'system wide site-packages dir not used on Mac OS X'
updated_at = <Date 2016-12-03.07:23:22.443>
user = 'https://bugs.python.org/kapet'

bugs.python.org fields:

activity = <Date 2016-12-03.07:23:22.443>
actor = 'ned.deily'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2009-03-30.23:16:33.941>
closer = 'ronaldoussoren'
components = ['Library (Lib)', 'macOS']
creation = <Date 2009-01-07.12:51:01.510>
creator = 'kapet'
dependencies = []
files = ['12637']
hgrepos = []
issue_num = 4865
keywords = ['patch']
message_count = 7.0
messages = ['79329', '79352', '79355', '84552', '84711', '282273', '282276']
nosy_count = 5.0
nosy_names = ['loewis', 'ronaldoussoren', 'ned.deily', 'kapet', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4865'
versions = ['Python 3.1', 'Python 2.7']

[kapet]

[Guido asked me to file this as an issue.]

On Mac OS X Apple has configured the system provided Python to also
search this global site-packages directory:
/Library/Python/<version>/site-packages/

This allows users to install their own system-wide packages there (and
not have them to drop into /System/Library/Frameworks/Python....../).

Unfortunately if you install Python from python.org (or pythonmac.org)
it is not searching this path, it only searches the site-packages
directory _within_ its own framework:
/Library/Frameworks/Python...../
That directory is of course not searched by the Apple provided Python.

This is an issue for addon package installers (think PIL or numpy):
There is no single path that will be found by both the system provided
Python and a Python installed from python.org. This makes most/all (?)
of the pre-built packages on pythonmac.org unusable with the version of
Python Apple ships although there is no version conflict or such - it
simply installs stuff in the "wrong" directories.

Many people who tried to install PIL for Django or App Engine stumbled
into this. The system provided Python version was fine. But compiling
PIL is hard. So they installed PIL from pythonmac.org. That couldn't
be found by Python. Now some people manually copied stuff around, some
created symlinks, some installed a second Python 2.5. (The lucky ones
picked a Python download from python.org, the not so lucky ones
installed the 2.5.0 release from pythonmac.org...)

I've attached a diff of a stock Python 2.5.1 library against what I
found on my Apple Mac OS X 10.5 (Leopard) system. I've dropped changes
that are not related to the path.

[loewis]

IIUC, you are not the author of the patch (in the sense that you wrote
the new code), but rather created a diff between Apple's code and
Python's. In this case, we cannot accept the patch - it needs to be
contributed by the authors (i.e. Apple).

Furthermore, I fail to see how this patch can affect the places were
Python searches for site-packages - it only deals with Python code that
might need adjustment if Python would search in this other location.

Please correct me if I'm wrong.

[kapet]

Sorry, I should have made this clear: I am NOT proposing to add this
patch. I meant this more to show what related changes were done by
Apple. Some of these clearly wouldn't even work like this if applied to
a Python not installed in /System/Library/Frameworks/...

IMHO the most interesting piece of that diff is the change to site.py
which makes it look into /Library/Python/

[ronaldoussoren]

I'm in favour of adding /Library/Python/x.y/ to sys.path for Python 2.7
and 3.1 and will work on that during the Pycon sprints.

/Library/Python will be added after the site-packages directory inside
the framework instead of replacing the latter directory. The reason for
this is that we don't want to install all packages in a location that's
visible to Apple's copy of Python because you could accidently replace
packages that Apple ships as part of the "Extras" directory that way.
Replacing those is bad because those libraries are used by Apple system
software and replacing stuff might therefore break the system
(particularly on server systems).

I won't backport to 2.6 and 3.0 because this is a feature request and
not a bugfix.

W.r.t. building PIL and other extensions: we've had a discussion about
this during the startup phase of the macpython-sig sprint and we have
some ideas to make this situation less painful in the future (without
necessarily having to patch Python for that).

[ronaldoussoren]

Committed a fix for this as r70778 (trunk), r70782 (3.1)

[python-dev]

New changeset a8a342b3fbc7 by Ned Deily in branch '2.7':
Issue bpo-28440: No longer add /Library/Python/site-packages, the Apple-supplied
https://hg.python.org/cpython/rev/a8a342b3fbc7

[ned-deily]

Update: this change has been reverted in Python 2.7.13. See Issue bpo-28440 for details.