deal with OSX "Extras" preventing upgrades

[netheril96]

My operating system is OS X Yosemite with system python. I installed pip with sudo easy_install pip. When trying to install latest packages into the user site, it always report a fake success.

For example:

pip install --user --upgrade numpy

Collecting numpy from https://pypi.python.org/packages/cp27/n/numpy/numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#md5=296f576bb648b8195b379b0bf39791ce
  Using cached numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: numpy

Successfully installed numpy-1.8.0rc1

So basically it downloads 1.9.2 and tells me it successfully upgrades to 1.8.0rc1. I also tried add an option --ignore-installed, but the same thing happens.

[piotr-dobrogost]

It might be the same problem as in #1018. What's the output of python -c 'import sys; print sys.path'?

[netheril96]

@piotr-dobrogost

The contents of sys.path is

['',
 '/Library/Python/2.7/site-packages/pip-6.0.8-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/rsy/Library/Python/2.7/lib/python/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages'
]

So by specifying the environment variable PYTHONPATH to $HOME/Library/Python/2.7/lib/python/site-packages I am able to actually upgrade numpy. But it doesn't work with GUI applications. It is also odd that this isn't in the front of sys.path by default.

[piotr-dobrogost]

So by specifying the environment variable PYTHONPATH I am able to actually upgrade numpy

What happens is that pip actually does upgrade numpy to version 1.9.2 (in user site-packages) but reports version of numpy which it finds looking at sys.path and this apparently is version 1.8.0rc1. By modifing PYTHONPATH you force Python to use newly installed version. Where is version 1.8.0rc1 installed? I guess it's in one of directories in sys.path which come before '/Users/rsy/Library/Python/2.7/lib/python/site-packages'. How was numpy version 1.8.0rc1 installed? Btw, using sudo pip (...) is almost always a bad idea. See #1668 for details.

[netheril96]

@piotr-dobrogost

numpy 1.8.0rc1 comes preinstalled with OS X. It is in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python. Still, why isn't the order of sys.path in this way? Shouldn't the user specific site take precedence? Any suggest fix?

[piotr-dobrogost]

User site directory is defined in PEP 370 where we read:

The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH . This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidently overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH .

It looks like on Mac, directory /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python has status of Python's search path (judging by its placement among other /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/* directories). It seems strange for a directory called Extras to be one of Python's search paths as these paths are normally paths to Python's core components. Maybe someone with knowledge of Python on Mac like @ned-deily could say more (quick search reveals someone asked exactly this question at Stackoverflow - http://stackoverflow.com/q/14881205/95735). However it is as it is and the only option (besides directly modifying sys.path) to override packages from Extras directory is to use PYTHONPATH environment variable. Having said this I suggest you try virtualenv instead.

[ned-deily]

Talk to Apple about that. The Extras directories are something they added to their system Pythons when they started shipping a number of third-party Python packages with the system Pythons in OS X releases. Extras is not part of the standard Python distribution nor other Python distributions on OS X, like the python.org binary installers. Also, note that Apple continues to ship very old versions of setuptools and easy_install with their system Pythons and no pip.

[netheril96]

@ned-deily @piotr-dobrogost

homebrew has problems with virtualenv, so I would rather stick with a vanilla one. Is there any configuration files I can edit to change the order of sys.path?

[piotr-dobrogost]

Yes, you can use .pth file for this. See http://stackoverflow.com/a/10739838/95735

[netheril96]

@piotr-dobrogost

It seems that .pth file would append the path to sys.path. In fact, in my sys.path, there is one item /usr/local/lib/python2.7/site-packages introduced through .pth file (even though that is a standard search path on Linux). It is still going to be masked by the older, system version of packages.

[piotr-dobrogost]

Right. You need to abuse .pth file the same way setuptools does; by inserting some Python code there. See easy_install.pth file, which is created and managed by setuptools, as an example.

[netheril96]

Can this be fixed or patched in the pip installation? New users of python on Mac should not learn how to work around this bug. They would probably just go for the easy sudo solution, which is dangerous.

[ned-deily]

One option is to not use the Apple-supplied Python (2.7.6 for OS X 10.10) but install a newer one: for example, the current python.org 2.7.9 or 3.4.3 installers for OS X, both of which now also install versions of pip. Or install newer versions of Python and pip using Homebrew, MacPorts, Anaconda, or other third-party distributors. It's unfortunate that Apple is still shipping the ancient setuptools version and such a non-standard site packages setup.

Another, non-standard option is to look at Glyph's pip2014 project (https://pip2014.com), a patched version of pip and virtualenv which, among other things, tries to workaround the Apple setuptools problem. I don't have any personal experience using it and I don't know what the pip developers think about it.

[dstufft]

pip2014 is generally good stuff, and we have open issues to try and obsolete it. One of them is what to do about the fact that Apple puts it's own third party stuff in front of --user and in front of --global. It's a non obvious problem, because on one hand we don't want to be responsible for fixing every way that any random platform decides to break their Python installations, but on the other we generally want things to work.

One option I've thought about is modifying the get-pip.py script to drop the needed .pth file, though that wouldn't help with this issue itself so that might not be a good enough suggestion.

[piotr-dobrogost]

@dstufft

Is Apple aware (is there any public bug) that the way they are injecting Extras directory is plainly wrong and hindrance for users? I would say make it sure they know it's broken and leave it as is. This way there's at least some pressure to fix it.

@netheril96

The simplest and cleanest way to fix this is by setting PYTHONPATH. You mentioned earlier that it does not work for GUI apps and I guess this is the reason why you're looking for other solution. I don't see why this wouldn't work for GUI apps so I would suggest to take a look at this problem as this might be easier to fix.

[netheril96]

@piotr-dobrogost

Setting PYTHONPATH does work for GUI apps, but it is very convoluted to do so on the Mac. .profile or .bashrc/.zshrc only work for executables launched with the shell.

Anyway, I personally solve this problem by switching to brewed python. But perhaps this issue should not be closed before a better solution for the general populace is found?

[rbtcollins]

So --user --upgrade is also a problem on Ubuntu too (and they default to --user now) - can we make the title a little broader? Something like:

packages that are installed in shadowed locations do not warn or error (e.g. OS X Extras, Ubuntu patched pip)

[Pablo-Arias]

I have the same problem with scipy, even if I upgrade to a newer version with pip and everything goes fine, when i try

import scipy
scipy.__version

I still have the old version, before the update...nothing changes....

But pip list does see the new scipy version... Any hints?

I'm on Yosemite 10.10.3

[dstufft]

This is solved now with macOS Sierra.

[piotr-dobrogost]

@dstufft
Can you provide some pointer to information how it was fixed?

[dstufft]

Apple stopped forcing the Extras directory to be before everything and instead added an Extras.pth in site-packages that adds it at the end.

[jeffoneill]

I'm on Sierra, and I didn't have an Extras.pth in site-packages, but I was still getting "Extras" early in my Python path. My site-packages was cluttered with tons of stuff over many years so I decided to wipe it all out and start over again. I don't know why, but that fixed my problem and I no longer have Extras on my path at all. Posting here in case it helps others.