Installing ipython on MacOS X tries to install to /System

[schwa]

Trying to install ipython "out of the box" on Mac OS X 10.7 using pip fails

It looks like it's trying to install files into "/System/Library/Frameworks/Python.framework/Versions/2.7/share"

There's no reason anything should be installed in there by pip.

Related? #295 and ipython/ipython#1231 (comment)

[schwa@ungoliant] ~$ pip install ipython
Downloading/unpacking ipython
  Running setup.py egg_info for package ipython

    no previously-included directories found matching 'IPython/deathrow'
    no previously-included directories found matching 'IPython/frontend/html/notebook/static/mathjax'
    warning: no previously-included files found matching 'docs/#*'
    warning: no previously-included files found matching 'docs/man/*.1'
    no previously-included directories found matching 'docs/attic'
    no previously-included directories found matching 'docs/build'
    no previously-included directories found matching 'docs/gh-pages'
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.flc' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    warning: no previously-included files matching '.dircopy.log' found anywhere in distribution
Installing collected packages: ipython
  Running setup.py install for ipython
    error: could not create '/System/Library/Frameworks/Python.framework/Versions/2.7/share': Permission denied
    Complete output from command /usr/bin/python -c "import setuptools;__file__='/Users/schwa/build/ipython/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /var/folders/c9/pqvs14g940j9q5dxgh8njsnh0000gn/T/pip-eBq04h-record/install-record.txt:
    running install

running build

running build_py

running install_lib

copying build/lib/IPython/.git_commit_info.ini -> /Library/Python/2.7/site-packages/IPython

running install_data

creating /System/Library/Frameworks/Python.framework/Versions/2.7/share

error: could not create '/System/Library/Frameworks/Python.framework/Versions/2.7/share': Permission denied

----------------------------------------
Command /usr/bin/python -c "import setuptools;__file__='/Users/schwa/build/ipython/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --single-version-externally-managed --record /var/folders/c9/pqvs14g940j9q5dxgh8njsnh0000gn/T/pip-eBq04h-record/install-record.txt failed with error code 1
Storing complete log in /Users/schwa/Library/Logs/pip.log
[schwa@ungoliant] ~$ 

[schwa]

pip install --user ipython does work

[carljm]

ipython uses the data_files option in setup.py with relative paths. According to the distutils documentation, such files will be installed relative to sys.prefix or sys.exec_prefix. I'm not deeply familiar with the oddities of OS X framework builds of Python, but I think it's quite likely that /System/Library/Frameworks/Python.framework/Versions/2.7/ is in fact sys.prefix on your Python. So it's not correct to say "there's no reason anything should be installed in there by pip" - that's precisely the location where IPython's setup.py is requesting for those files to be installed. (On Linux system installations, sys.prefix is typically /usr, which would result in the more sane location of /usr/share. The seeming strangeness of the location you're seeing is just a result of distutils assumptions interacting poorly with OS X framework build wierdness.) Pip isn't doing anything different with these files than what any other installation mechanism would do.

Regardless, it appears to me that what you're seeing is a straight-up permissions failure which likely would have caused the installation to fail anyway, even in the absence of these data_files, as soon as pip/setuptools attempted to install the Python files themselves, since you're trying to do a system-wide install without sudo. (Unless you have an odd permissions setup where your user has perms for the system Python's site-packages directory, but not for sys.prefix.)

pip install --user ipython works, of course, because it installs everything under the ~/.local prefix, which you of course have permissions to write to.

Thanks for the report, but I don't see any bug in pip here, closing.

[schwa]

Well users (and developers) are strongly encouraged to not write to /System/Library, Apple effectively "owns" that directory and any system update can and probably will stomp all over any user added files.

You're quite correct sys.prefix is pointing there:

sys.prefix
'/System/Library/Frameworks/Python.framework/Versions/2.7'

However the actual packages installed by pip go into /Library (in my case /Library/Python/2.7/site-packages) just fine (which the user is in control of and shouldn't be overwritten in system update).

I'm not sure why packages go into /Library and package data into /System/Library

Obviously something is in conflict here and I'm not enough of a python distutils expert to know what. However I do know that easy_install does seem to do the right thing.

What this does mean is that users with the OS built-in python are unable to install ipython (and other packages that use the same feature) without a sudo. And if they do use sudo they're installing parts of the ipython to a directory that can and will get overridden by an OS system update.

[schwa]

I think this issue should be reopened (it was closed while I was writing my follow up comment).

While yes, technically pip is just doing what it thinks it should be doing, it is trying (although failing due to permissions)
to install files in a location it should not be touching.

This potentially affects anyone who is trying to use pip with the system installed version of python (will like to test hand installed pythons too).

[carljm]

What's in conflict here, as I said before, is distutils assumptions around data_files and the unusual way Apple lays out their OS X framework builds of Python.

The reason easy_install works is because it doesn't follow the distutils standard way of installing, it installs everything related to a package into a single egg directory under site-packages. So with easy_install the share directories don't end up where they'd be expected (e.g. /usr/share on Linux) but rather in e.g. /usr/lib/python2.7/site-packages/ipython-0.12-py2.7.egg/share. It happens that this plays nicer with the way Apple sets up the directories and permissions for framework builds, but what pip is doing is the way non-egg installations have always worked with data_files.

This is something that will only be fixed by either Apple fixing the layout of framework builds to match more closely how it works on other platforms, or distutils changing the documented behavior of data_files (which is happening in the work on distutils2). In the meantime, even if we wanted to add a workaround to pip to special-case OS X framework builds, it would be prohibitively difficult to do so, as pip doesn't perform the actual installation itself or find any of these paths itself, it delegates to setuptools/distutils for that. And setuptools/distutils is doing exactly what it has always done and is documented to do.

[schwa]

Alright. Fair enough. Not your bug, even if it rather dramatically limits the usefulness of pip.

I'll see where I can file this bug next…

[carljm]

If you want to see this bug without involving pip at all, try running a regular python setup.py install on a git checkout or unpacked sdist of ipython; you'll see the same behavior.

(My recommendation, FWIW, is for ipython and all other packages to simply avoid the use of data_files until distutils2 brings a more sophisticated data-files mechanism into the Python ecosystem.)

[akvadrako]

The workaround (at least for ipython) is to add these lines to ~/.pydistutils.cfg:

[install]
install-data=/usr/local
install-scripts=/usr/local/bin

[SamB]

So, basically, this bug is in the cpython tree?

[carljm]

@SamB Well, it's a bad interaction between a not-very-sophisticated feature in distutils (data_files), which is in the CPython tree, and Apple's choice to lay out OS X system installs of Python differently from any other Python installation, and in a way that violates distutils' assumption (legitimate with an unmodified Python build) that site-packages is located beneath sys.prefix. So you can take your pick of where you want to lay the blame there ;-)

The most likely path to getting the bug fixed in a reasonable time frame would actually be for ipython's setup.py to detect OS X framework builds and leave out the data_files kwarg in that case. That would be quite easy to do.