deal with OSX "Extras" preventing upgrades #2468

Closed
netheril96 opened this Issue Mar 2, 2015 · 21 comments

Comments

Projects
None yet
8 participants
@netheril96

My operating system is OS X Yosemite with system python. I installed pip with sudo easy_install pip. When trying to install latest packages into the user site, it always report a fake success.

For example:

pip install --user --upgrade numpy

Collecting numpy from https://pypi.python.org/packages/cp27/n/numpy/numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#md5=296f576bb648b8195b379b0bf39791ce
  Using cached numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: numpy

Successfully installed numpy-1.8.0rc1

So basically it downloads 1.9.2 and tells me it successfully upgrades to 1.8.0rc1. I also tried add an option --ignore-installed, but the same thing happens.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 2, 2015

It might be the same problem as in #1018. What's the output of python -c 'import sys; print sys.path'?

It might be the same problem as in #1018. What's the output of python -c 'import sys; print sys.path'?

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Mar 2, 2015

@piotr-dobrogost

The contents of sys.path is

['',
 '/Library/Python/2.7/site-packages/pip-6.0.8-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/rsy/Library/Python/2.7/lib/python/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages'
]

So by specifying the environment variable PYTHONPATH to $HOME/Library/Python/2.7/lib/python/site-packages I am able to actually upgrade numpy. But it doesn't work with GUI applications. It is also odd that this isn't in the front of sys.path by default.

@piotr-dobrogost

The contents of sys.path is

['',
 '/Library/Python/2.7/site-packages/pip-6.0.8-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/rsy/Library/Python/2.7/lib/python/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages'
]

So by specifying the environment variable PYTHONPATH to $HOME/Library/Python/2.7/lib/python/site-packages I am able to actually upgrade numpy. But it doesn't work with GUI applications. It is also odd that this isn't in the front of sys.path by default.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 2, 2015

So by specifying the environment variable PYTHONPATH I am able to actually upgrade numpy

What happens is that pip actually does upgrade numpy to version 1.9.2 (in user site-packages) but reports version of numpy which it finds looking at sys.path and this apparently is version 1.8.0rc1. By modifing PYTHONPATH you force Python to use newly installed version. Where is version 1.8.0rc1 installed? I guess it's in one of directories in sys.path which come before '/Users/rsy/Library/Python/2.7/lib/python/site-packages'. How was numpy version 1.8.0rc1 installed? Btw, using sudo pip (...) is almost always a bad idea. See #1668 for details.

So by specifying the environment variable PYTHONPATH I am able to actually upgrade numpy

What happens is that pip actually does upgrade numpy to version 1.9.2 (in user site-packages) but reports version of numpy which it finds looking at sys.path and this apparently is version 1.8.0rc1. By modifing PYTHONPATH you force Python to use newly installed version. Where is version 1.8.0rc1 installed? I guess it's in one of directories in sys.path which come before '/Users/rsy/Library/Python/2.7/lib/python/site-packages'. How was numpy version 1.8.0rc1 installed? Btw, using sudo pip (...) is almost always a bad idea. See #1668 for details.

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Mar 2, 2015

@piotr-dobrogost

numpy 1.8.0rc1 comes preinstalled with OS X. It is in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python. Still, why isn't the order of sys.path in this way? Shouldn't the user specific site take precedence? Any suggest fix?

@piotr-dobrogost

numpy 1.8.0rc1 comes preinstalled with OS X. It is in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python. Still, why isn't the order of sys.path in this way? Shouldn't the user specific site take precedence? Any suggest fix?

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 2, 2015

User site directory is defined in PEP 370 where we read:

The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH . This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidently overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH .

It looks like on Mac, directory /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python has status of Python's search path (judging by its placement among other /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/* directories). It seems strange for a directory called Extras to be one of Python's search paths as these paths are normally paths to Python's core components. Maybe someone with knowledge of Python on Mac like @ned-deily could say more (quick search reveals someone asked exactly this question at Stackoverflow - http://stackoverflow.com/q/14881205/95735). However it is as it is and the only option (besides directly modifying sys.path) to override packages from Extras directory is to use PYTHONPATH environment variable. Having said this I suggest you try virtualenv instead.

User site directory is defined in PEP 370 where we read:

The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH . This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidently overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH .

It looks like on Mac, directory /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python has status of Python's search path (judging by its placement among other /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/* directories). It seems strange for a directory called Extras to be one of Python's search paths as these paths are normally paths to Python's core components. Maybe someone with knowledge of Python on Mac like @ned-deily could say more (quick search reveals someone asked exactly this question at Stackoverflow - http://stackoverflow.com/q/14881205/95735). However it is as it is and the only option (besides directly modifying sys.path) to override packages from Extras directory is to use PYTHONPATH environment variable. Having said this I suggest you try virtualenv instead.

@ned-deily

This comment has been minimized.

Show comment
Hide comment
@ned-deily

ned-deily Mar 2, 2015

Talk to Apple about that. The Extras directories are something they added to their system Pythons when they started shipping a number of third-party Python packages with the system Pythons in OS X releases. Extras is not part of the standard Python distribution nor other Python distributions on OS X, like the python.org binary installers. Also, note that Apple continues to ship very old versions of setuptools and easy_install with their system Pythons and no pip.

Talk to Apple about that. The Extras directories are something they added to their system Pythons when they started shipping a number of third-party Python packages with the system Pythons in OS X releases. Extras is not part of the standard Python distribution nor other Python distributions on OS X, like the python.org binary installers. Also, note that Apple continues to ship very old versions of setuptools and easy_install with their system Pythons and no pip.

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Mar 3, 2015

@ned-deily @piotr-dobrogost

homebrew has problems with virtualenv, so I would rather stick with a vanilla one. Is there any configuration files I can edit to change the order of sys.path?

@ned-deily @piotr-dobrogost

homebrew has problems with virtualenv, so I would rather stick with a vanilla one. Is there any configuration files I can edit to change the order of sys.path?

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 3, 2015

Yes, you can use .pth file for this. See http://stackoverflow.com/a/10739838/95735

Yes, you can use .pth file for this. See http://stackoverflow.com/a/10739838/95735

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Mar 3, 2015

@piotr-dobrogost

It seems that .pth file would append the path to sys.path. In fact, in my sys.path, there is one item /usr/local/lib/python2.7/site-packages introduced through .pth file (even though that is a standard search path on Linux). It is still going to be masked by the older, system version of packages.

@piotr-dobrogost

It seems that .pth file would append the path to sys.path. In fact, in my sys.path, there is one item /usr/local/lib/python2.7/site-packages introduced through .pth file (even though that is a standard search path on Linux). It is still going to be masked by the older, system version of packages.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 3, 2015

Right. You need to abuse .pth file the same way setuptools does; by inserting some Python code there. See easy_install.pth file, which is created and managed by setuptools, as an example.

Right. You need to abuse .pth file the same way setuptools does; by inserting some Python code there. See easy_install.pth file, which is created and managed by setuptools, as an example.

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Mar 4, 2015

Can this be fixed or patched in the pip installation? New users of python on Mac should not learn how to work around this bug. They would probably just go for the easy sudo solution, which is dangerous.

Can this be fixed or patched in the pip installation? New users of python on Mac should not learn how to work around this bug. They would probably just go for the easy sudo solution, which is dangerous.

@ned-deily

This comment has been minimized.

Show comment
Hide comment
@ned-deily

ned-deily Mar 4, 2015

One option is to not use the Apple-supplied Python (2.7.6 for OS X 10.10) but install a newer one: for example, the current python.org 2.7.9 or 3.4.3 installers for OS X, both of which now also install versions of pip. Or install newer versions of Python and pip using Homebrew, MacPorts, Anaconda, or other third-party distributors. It's unfortunate that Apple is still shipping the ancient setuptools version and such a non-standard site packages setup.

Another, non-standard option is to look at Glyph's pip2014 project (https://pip2014.com), a patched version of pip and virtualenv which, among other things, tries to workaround the Apple setuptools problem. I don't have any personal experience using it and I don't know what the pip developers think about it.

One option is to not use the Apple-supplied Python (2.7.6 for OS X 10.10) but install a newer one: for example, the current python.org 2.7.9 or 3.4.3 installers for OS X, both of which now also install versions of pip. Or install newer versions of Python and pip using Homebrew, MacPorts, Anaconda, or other third-party distributors. It's unfortunate that Apple is still shipping the ancient setuptools version and such a non-standard site packages setup.

Another, non-standard option is to look at Glyph's pip2014 project (https://pip2014.com), a patched version of pip and virtualenv which, among other things, tries to workaround the Apple setuptools problem. I don't have any personal experience using it and I don't know what the pip developers think about it.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Mar 4, 2015

Member

pip2014 is generally good stuff, and we have open issues to try and obsolete it. One of them is what to do about the fact that Apple puts it's own third party stuff in front of --user and in front of --global. It's a non obvious problem, because on one hand we don't want to be responsible for fixing every way that any random platform decides to break their Python installations, but on the other we generally want things to work.

One option I've thought about is modifying the get-pip.py script to drop the needed .pth file, though that wouldn't help with this issue itself so that might not be a good enough suggestion.

Member

dstufft commented Mar 4, 2015

pip2014 is generally good stuff, and we have open issues to try and obsolete it. One of them is what to do about the fact that Apple puts it's own third party stuff in front of --user and in front of --global. It's a non obvious problem, because on one hand we don't want to be responsible for fixing every way that any random platform decides to break their Python installations, but on the other we generally want things to work.

One option I've thought about is modifying the get-pip.py script to drop the needed .pth file, though that wouldn't help with this issue itself so that might not be a good enough suggestion.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Mar 4, 2015

@dstufft

Is Apple aware (is there any public bug) that the way they are injecting Extras directory is plainly wrong and hindrance for users? I would say make it sure they know it's broken and leave it as is. This way there's at least some pressure to fix it.

@netheril96

The simplest and cleanest way to fix this is by setting PYTHONPATH. You mentioned earlier that it does not work for GUI apps and I guess this is the reason why you're looking for other solution. I don't see why this wouldn't work for GUI apps so I would suggest to take a look at this problem as this might be easier to fix.

@dstufft

Is Apple aware (is there any public bug) that the way they are injecting Extras directory is plainly wrong and hindrance for users? I would say make it sure they know it's broken and leave it as is. This way there's at least some pressure to fix it.

@netheril96

The simplest and cleanest way to fix this is by setting PYTHONPATH. You mentioned earlier that it does not work for GUI apps and I guess this is the reason why you're looking for other solution. I don't see why this wouldn't work for GUI apps so I would suggest to take a look at this problem as this might be easier to fix.

@netheril96

This comment has been minimized.

Show comment
Hide comment
@netheril96

netheril96 Mar 4, 2015

@piotr-dobrogost

Setting PYTHONPATH does work for GUI apps, but it is very convoluted to do so on the Mac. .profile or .bashrc/.zshrc only work for executables launched with the shell.

Anyway, I personally solve this problem by switching to brewed python. But perhaps this issue should not be closed before a better solution for the general populace is found?

@piotr-dobrogost

Setting PYTHONPATH does work for GUI apps, but it is very convoluted to do so on the Mac. .profile or .bashrc/.zshrc only work for executables launched with the shell.

Anyway, I personally solve this problem by switching to brewed python. But perhaps this issue should not be closed before a better solution for the general populace is found?

@qwcode qwcode changed the title from `pip install --user --upgrade` reports success, but does not actually upgrade to deal with OSX "Extras" preventing upgrades Mar 5, 2015

@ctberthiaume ctberthiaume referenced this issue in ctberthiaume/gdcp Apr 6, 2015

Closed

install problem #11

@curita curita referenced this issue in scrapy/scrapy Jul 10, 2015

Closed

Can't install on OSX 10.10.4 #1342

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Sep 9, 2015

Contributor

So --user --upgrade is also a problem on Ubuntu too (and they default to --user now) - can we make the title a little broader? Something like:

packages that are installed in shadowed locations do not warn or error (e.g. OS X Extras, Ubuntu patched pip)

Contributor

rbtcollins commented Sep 9, 2015

So --user --upgrade is also a problem on Ubuntu too (and they default to --user now) - can we make the title a little broader? Something like:

packages that are installed in shadowed locations do not warn or error (e.g. OS X Extras, Ubuntu patched pip)

@Pablo-Arias

This comment has been minimized.

Show comment
Hide comment
@Pablo-Arias

Pablo-Arias May 27, 2016

I have the same problem with scipy, even if I upgrade to a newer version with pip and everything goes fine, when i try

import scipy
scipy.__version

I still have the old version, before the update...nothing changes....

But pip list does see the new scipy version... Any hints?

I'm on Yosemite 10.10.3

Pablo-Arias commented May 27, 2016

I have the same problem with scipy, even if I upgrade to a newer version with pip and everything goes fine, when i try

import scipy
scipy.__version

I still have the old version, before the update...nothing changes....

But pip list does see the new scipy version... Any hints?

I'm on Yosemite 10.10.3

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Sep 22, 2016

Member

This is solved now with macOS Sierra.

Member

dstufft commented Sep 22, 2016

This is solved now with macOS Sierra.

@dstufft dstufft closed this Sep 22, 2016

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Sep 23, 2016

@dstufft
Can you provide some pointer to information how it was fixed?

@dstufft
Can you provide some pointer to information how it was fixed?

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Sep 23, 2016

Member

Apple stopped forcing the Extras directory to be before everything and instead added an Extras.pth in site-packages that adds it at the end.

Member

dstufft commented Sep 23, 2016

Apple stopped forcing the Extras directory to be before everything and instead added an Extras.pth in site-packages that adds it at the end.

@jeffoneill

This comment has been minimized.

Show comment
Hide comment
@jeffoneill

jeffoneill Sep 3, 2017

I'm on Sierra, and I didn't have an Extras.pth in site-packages, but I was still getting "Extras" early in my Python path. My site-packages was cluttered with tons of stuff over many years so I decided to wipe it all out and start over again. I don't know why, but that fixed my problem and I no longer have Extras on my path at all. Posting here in case it helps others.

I'm on Sierra, and I didn't have an Extras.pth in site-packages, but I was still getting "Extras" early in my Python path. My site-packages was cluttered with tons of stuff over many years so I decided to wipe it all out and start over again. I don't know why, but that fixed my problem and I no longer have Extras on my path at all. Posting here in case it helps others.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment