Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deal with OSX "Extras" preventing upgrades #2468

Closed
netheril96 opened this issue Mar 2, 2015 · 21 comments
Closed

deal with OSX "Extras" preventing upgrades #2468

netheril96 opened this issue Mar 2, 2015 · 21 comments

Comments

@netheril96
Copy link

@netheril96 netheril96 commented Mar 2, 2015

My operating system is OS X Yosemite with system python. I installed pip with sudo easy_install pip. When trying to install latest packages into the user site, it always report a fake success.

For example:

pip install --user --upgrade numpy

Collecting numpy from https://pypi.python.org/packages/cp27/n/numpy/numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl#md5=296f576bb648b8195b379b0bf39791ce
  Using cached numpy-1.9.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Installing collected packages: numpy

Successfully installed numpy-1.8.0rc1

So basically it downloads 1.9.2 and tells me it successfully upgrades to 1.8.0rc1. I also tried add an option --ignore-installed, but the same thing happens.

@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Mar 2, 2015

It might be the same problem as in #1018. What's the output of python -c 'import sys; print sys.path'?

@netheril96
Copy link
Author

@netheril96 netheril96 commented Mar 2, 2015

@piotr-dobrogost

The contents of sys.path is

['',
 '/Library/Python/2.7/site-packages/pip-6.0.8-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/rsy/Library/Python/2.7/lib/python/site-packages',
 '/usr/local/lib/python2.7/site-packages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages'
]

So by specifying the environment variable PYTHONPATH to $HOME/Library/Python/2.7/lib/python/site-packages I am able to actually upgrade numpy. But it doesn't work with GUI applications. It is also odd that this isn't in the front of sys.path by default.

@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Mar 2, 2015

So by specifying the environment variable PYTHONPATH I am able to actually upgrade numpy

What happens is that pip actually does upgrade numpy to version 1.9.2 (in user site-packages) but reports version of numpy which it finds looking at sys.path and this apparently is version 1.8.0rc1. By modifing PYTHONPATH you force Python to use newly installed version. Where is version 1.8.0rc1 installed? I guess it's in one of directories in sys.path which come before '/Users/rsy/Library/Python/2.7/lib/python/site-packages'. How was numpy version 1.8.0rc1 installed? Btw, using sudo pip (...) is almost always a bad idea. See #1668 for details.

@netheril96
Copy link
Author

@netheril96 netheril96 commented Mar 2, 2015

@piotr-dobrogost

numpy 1.8.0rc1 comes preinstalled with OS X. It is in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python. Still, why isn't the order of sys.path in this way? Shouldn't the user specific site take precedence? Any suggest fix?

@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Mar 2, 2015

User site directory is defined in PEP 370 where we read:

The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH . This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidently overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH .

It looks like on Mac, directory /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python has status of Python's search path (judging by its placement among other /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/* directories). It seems strange for a directory called Extras to be one of Python's search paths as these paths are normally paths to Python's core components. Maybe someone with knowledge of Python on Mac like @ned-deily could say more (quick search reveals someone asked exactly this question at Stackoverflow - http://stackoverflow.com/q/14881205/95735). However it is as it is and the only option (besides directly modifying sys.path) to override packages from Extras directory is to use PYTHONPATH environment variable. Having said this I suggest you try virtualenv instead.

@ned-deily
Copy link

@ned-deily ned-deily commented Mar 2, 2015

Talk to Apple about that. The Extras directories are something they added to their system Pythons when they started shipping a number of third-party Python packages with the system Pythons in OS X releases. Extras is not part of the standard Python distribution nor other Python distributions on OS X, like the python.org binary installers. Also, note that Apple continues to ship very old versions of setuptools and easy_install with their system Pythons and no pip.

@netheril96
Copy link
Author

@netheril96 netheril96 commented Mar 3, 2015

@ned-deily @piotr-dobrogost

homebrew has problems with virtualenv, so I would rather stick with a vanilla one. Is there any configuration files I can edit to change the order of sys.path?

@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Mar 3, 2015

Yes, you can use .pth file for this. See http://stackoverflow.com/a/10739838/95735

@netheril96
Copy link
Author

@netheril96 netheril96 commented Mar 3, 2015

@piotr-dobrogost

It seems that .pth file would append the path to sys.path. In fact, in my sys.path, there is one item /usr/local/lib/python2.7/site-packages introduced through .pth file (even though that is a standard search path on Linux). It is still going to be masked by the older, system version of packages.

@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Mar 3, 2015

Right. You need to abuse .pth file the same way setuptools does; by inserting some Python code there. See easy_install.pth file, which is created and managed by setuptools, as an example.

@netheril96
Copy link
Author

@netheril96 netheril96 commented Mar 4, 2015

Can this be fixed or patched in the pip installation? New users of python on Mac should not learn how to work around this bug. They would probably just go for the easy sudo solution, which is dangerous.

@ned-deily
Copy link

@ned-deily ned-deily commented Mar 4, 2015

One option is to not use the Apple-supplied Python (2.7.6 for OS X 10.10) but install a newer one: for example, the current python.org 2.7.9 or 3.4.3 installers for OS X, both of which now also install versions of pip. Or install newer versions of Python and pip using Homebrew, MacPorts, Anaconda, or other third-party distributors. It's unfortunate that Apple is still shipping the ancient setuptools version and such a non-standard site packages setup.

Another, non-standard option is to look at Glyph's pip2014 project (https://pip2014.com), a patched version of pip and virtualenv which, among other things, tries to workaround the Apple setuptools problem. I don't have any personal experience using it and I don't know what the pip developers think about it.

@dstufft
Copy link
Member

@dstufft dstufft commented Mar 4, 2015

pip2014 is generally good stuff, and we have open issues to try and obsolete it. One of them is what to do about the fact that Apple puts it's own third party stuff in front of --user and in front of --global. It's a non obvious problem, because on one hand we don't want to be responsible for fixing every way that any random platform decides to break their Python installations, but on the other we generally want things to work.

One option I've thought about is modifying the get-pip.py script to drop the needed .pth file, though that wouldn't help with this issue itself so that might not be a good enough suggestion.

@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Mar 4, 2015

@dstufft

Is Apple aware (is there any public bug) that the way they are injecting Extras directory is plainly wrong and hindrance for users? I would say make it sure they know it's broken and leave it as is. This way there's at least some pressure to fix it.

@netheril96

The simplest and cleanest way to fix this is by setting PYTHONPATH. You mentioned earlier that it does not work for GUI apps and I guess this is the reason why you're looking for other solution. I don't see why this wouldn't work for GUI apps so I would suggest to take a look at this problem as this might be easier to fix.

@netheril96
Copy link
Author

@netheril96 netheril96 commented Mar 4, 2015

@piotr-dobrogost

Setting PYTHONPATH does work for GUI apps, but it is very convoluted to do so on the Mac. .profile or .bashrc/.zshrc only work for executables launched with the shell.

Anyway, I personally solve this problem by switching to brewed python. But perhaps this issue should not be closed before a better solution for the general populace is found?

@qwcode qwcode changed the title `pip install --user --upgrade` reports success, but does not actually upgrade deal with OSX "Extras" preventing upgrades Mar 5, 2015
@rbtcollins
Copy link
Contributor

@rbtcollins rbtcollins commented Sep 9, 2015

So --user --upgrade is also a problem on Ubuntu too (and they default to --user now) - can we make the title a little broader? Something like:

packages that are installed in shadowed locations do not warn or error (e.g. OS X Extras, Ubuntu patched pip)

@Pablo-Arias
Copy link

@Pablo-Arias Pablo-Arias commented May 27, 2016

I have the same problem with scipy, even if I upgrade to a newer version with pip and everything goes fine, when i try

import scipy
scipy.__version

I still have the old version, before the update...nothing changes....

But pip list does see the new scipy version... Any hints?

I'm on Yosemite 10.10.3

@dstufft
Copy link
Member

@dstufft dstufft commented Sep 22, 2016

This is solved now with macOS Sierra.

@dstufft dstufft closed this Sep 22, 2016
@piotr-dobrogost
Copy link

@piotr-dobrogost piotr-dobrogost commented Sep 23, 2016

@dstufft
Can you provide some pointer to information how it was fixed?

@dstufft
Copy link
Member

@dstufft dstufft commented Sep 23, 2016

Apple stopped forcing the Extras directory to be before everything and instead added an Extras.pth in site-packages that adds it at the end.

@jeffoneill
Copy link

@jeffoneill jeffoneill commented Sep 3, 2017

I'm on Sierra, and I didn't have an Extras.pth in site-packages, but I was still getting "Extras" early in my Python path. My site-packages was cluttered with tons of stuff over many years so I decided to wipe it all out and start over again. I don't know why, but that fixed my problem and I no longer have Extras on my path at all. Posting here in case it helps others.

@lock lock bot added the S: auto-locked label Jun 3, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Jun 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.