Homebrew Python, Macvim: getting user scheme error (--user)

[oryband]

I'm using a Homebrew Python & Macvim, on OSX ML.
When trying to pip install --user git+git://github.com/Lokaltog/powerline, I get a user scheme related error:

------------------------------------------------------------
/usr/local/bin/pip run on Wed Jun 19 03:11:30 2013
Downloading/unpacking git+git://github.com/Lokaltog/powerline

  Cloning git://github.com/Lokaltog/powerline to /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build

  Found command 'git' at '/usr/local/bin/git'
  Running command /usr/local/bin/git clone -q git://github.com/Lokaltog/powerline /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build
  Running setup.py egg_info for package from git+git://github.com/Lokaltog/powerline

    running egg_info
    creating pip-egg-info/Powerline.egg-info
    writing requirements to pip-egg-info/Powerline.egg-info/requires.txt
    writing pip-egg-info/Powerline.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/Powerline.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/Powerline.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/Powerline.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found


    reading manifest file 'pip-egg-info/Powerline.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/Powerline.egg-info/SOURCES.txt'
  Source in /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build has version beta, which satisfies requirement Powerline==beta from git+git://github.com/Lokaltog/powerline
  skipping extra docs
Installing collected packages: Powerline

  Running setup.py install for Powerline

    Running command /usr/local/opt/python/bin/python2.7 -c "import setuptools;__file__='/var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-JsOX6p-record/install-record.txt --single-version-externally-managed --user
    running install
    error: can't combine user with prefix, exec_prefix/home, or install_(plat)base

    Complete output from command /usr/local/opt/python/bin/python2.7 -c "import setuptools;__file__='/var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-JsOX6p-record/install-record.txt --single-version-externally-managed --user:

    running install

error: can't combine user with prefix, exec_prefix/home, or install_(plat)base

----------------------------------------

Command /usr/local/opt/python/bin/python2.7 -c "import setuptools;__file__='/var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-JsOX6p-record/install-record.txt --single-version-externally-managed --user failed with error code 1 in /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build

Exception information:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/commands/install.py", line 271, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 1185, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/req.py", line 592, in install
    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pip-1.3.1-py2.7.egg/pip/util.py", line 662, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command /usr/local/opt/python/bin/python2.7 -c "import setuptools;__file__='/var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-JsOX6p-record/install-record.txt --single-version-externally-managed --user failed with error code 1 in /var/folders/k9/39bn572j64785jmvr63tqm180000gn/T/pip-CeX1GF-build

Any idea how to fix this?

[eddiemonge]

I am getting this as well now

[a-b]

me too

[kurtisnelson]

👍

[dsdeiz]

I got it working following this.

[oryband]

@dsdeiz This fixes the problem.

This should be stated in the docs.

[Lokaltog]

@oryband Feel free to add it and submit a pull request.

[a-b]

I wouldn't add an empty prefix as suggested at #552 (comment)

With empty prefix it will install packages to the ~/Library/Python/2.7/lib/python/site-packages/
Instead if you're using python installed by brew, you're expecting to have all your packages under /usr/local/lib/python2.7/site-packages as explained at brew info python

more info here https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python

[oryband]

@Lokaltog What do you suggest should be done?

[unphased]

@a-b What should we do?

I had to do this to get it installed, it sure looks like some stuff got dumped in ~/Library...

lust@Stevens-MacBook-Pro:~ 
$ echo "[install]\nprefix=\n" > ~/.pydistutils.cfg                                              
lust@Stevens-MacBook-Pro:~ 
$ pip install --user git+git://github.com/Lokaltog/powerline
Downloading/unpacking git+git://github.com/Lokaltog/powerline
  Cloning git://github.com/Lokaltog/powerline to /var/folders/z_/ybb_x1f96l9d2yqr18ncmg1w0000gn/T/pip-
i64Eno-build
  Running setup.py egg_info for package from git+git://github.com/Lokaltog/powerline

Installing collected packages: Powerline
  Running setup.py install for Powerline

    changing mode of build/scripts-2.7/powerline from 644 to 755
    changing mode of build/scripts-2.7/powerline-lint from 644 to 755
    changing mode of /Users/lust/Library/Python/2.7/bin/powerline to 755
    changing mode of /Users/lust/Library/Python/2.7/bin/powerline-lint to 755
Successfully installed Powerline
Cleaning up...

[gji]

In the Homebrew documentation at https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python:

Note, pip install --user is disabled for brewed Python. This is a bug in distutils, because Homebrew writes a distutils.cfg that sets the prefix for packages.

I installed powerline without the --user flag, which of course means that it is not installed in "/Users/..." but rather in "/usr/local/...." (or whatever your brew prefix is). According to the brew documentation, installing in ~/Library/Python/2.7/lib/python/site-packages/ will also work though.

[samueljohn]

So, if you use a brewed python (via brew install python), you can just do
pip install git+git://github.com/Lokaltog/powerline and the powerline module will be at $(brew --prefix)/lib/python2.7/site-packages. Don't need to set the PYTHONPATH.
But you have to make sure that you brew install vim after you brewed python, so that vim uses brewed python.
To check this, I use xcrun otool -L $(brew --prefix)/bin/vim and look for the linked Python.

Another option, to work around the "can't combine user with prefix, exec_prefix/home, or install_(plat)base" is to only disable the prefix temporarily:

pip install --user --install-option="--prefix=" git+git://github.com/Lokaltog/powerline

Note that this will put executable scripts into ~/Library/Python/2.7/bin.

[oryband]

This is too complicated.

[samueljohn]

I am even thinking to let homebrew patch distutils such that --user overwrites the prefix.

[Lokaltog]

Haha, this is ridiculous. I can't believe that OS X causes this many issues with installing a simple Python package. Is there any of this stuff that should be put into the docs? I can't test any of this myself so I'm relying on OS X users to provide correct installation instructions.

[samueljohn]

Wrong. This is no OS X issue. Not at all. Python's distutils should give --user precedence over the prefix set in a distutils.cfg file.

[Lokaltog]

I think the general Python user/system path mess is OS X specific (i.e. different Python paths depending on if Python is native or installed with brew, and different paths depending on if vim was compiled before or after Python had been installed with brew, and issues with --user not working). I've not been able to reproduce any of these issues on Linux.

It could of course also be a bug in distutils which is triggered in this scenario where --user is not given precedence over distutils.cfg.

[samueljohn]

I'd not say it is a mess. On Mac OS X, Python decided (not Apple) to put user related packages into ~/Library/Python/... instead of ~/.python or some other dot-directory. A global system wide site-packages is at the root /Library/Python. I think this is not bad.

[...] different Python paths depending on if Python is native or installed with brew

Again this is not true. Both system and brewed Python respect the above mentioned paths for packages.
Only thing, however, is that system Python does not know of the site-packages brew (and the pip that comes with a brewed python) installs to: $(brew --prefix)/lib/pythonX.Y/site-packages.

This is so due to several reasons, but mostly because homebrew promises not to write files outside of the brew --prefix and also because every python has a built-in location for site-packages at the sys.prefix. A brewed Python is no exception here.

What is indeed true is that Homebrew writes a distutils.cfg specifying the prefix for packages to be installed with distutils (so for pip) to be equal to the brew --prefix. Most linux don't do this - everything goes into /usr/local (I guess).

The only unfortunate thing is that distutils does not give more weight to --user. It is usually good practice that explicitly given flags overwrite some defaults from a configuration file.

Let me stress that --user is not necessary at all for a brewed python and vim:

$ brew install python
$ brew install vim
$ pip install git+git://github.com/Lokaltog/powerline

That is perfectly fine and I run my vim with powerline this way. Btw: Thanks for creating this software! 🍻

Without a brewed python, brew install vim will use system's Python:

$ which python
==> /usr/bin/python
$ brew install vim # will link against system python
# system python has no `pip` shipped, so
$ sudo easy_install pip
$ pip install --user git+git://github.com/Lokaltog/powerline

[Lokaltog]

Allright, I guess I've misunderstood some of the details surrounding this issue. Your instructions are clear so I think I'll update the OS X installation docs with those instead.

[samueljohn]

Cool, thanks!

[oryband]

So, what is the proposed fix? For now I still have a ~/.pydistutils.cfg which causes brew doctor to complain, but allows me to install powerline properly.

[samueljohn]

@oryband if you have a brewed python you can just leave out the --user part and still don't need sudo.

[oryband]

@samueljohn just updated brew, removed ~/.pydistutils.cfg and installed powerline without --user and I can confirm this works.

Thanks!

Please updated this in the docs and we can close this issue.

[derekbrokeit]

Just to point out for those saying this is OSX specific, it is not. It appears to be something with homebrew's installation of python. I ran into this error for multiple python packages. I have python installed using linuxbrew (homebrew for linux). (Note: I have modified the python installation formula for linux by stripping out all OSX/Framework specific components).

@samueljohn is correct that installation works flawlessly without --user, but in my particular case I need all packages installed as --user because my system has multiple python instances of python2.7. I have both homebrew python and a proprietary software package that essentially is a custom python distribution (incomplete and thus not linkable). Therefore, I need both. I fixed it by simply relying on the homebrew version of pip and making ~/.local/lib/python2.7/site-packages a soft-link to the homebrew-python site-packages directory. Not elegant, but worked. It also saves me time in the end.