pip install --user misbehaves inside venv

[raffomania]

Environment

• pip version: 18.0
• Python version: 3.7
• OS: Arch Linux, updated on 12. August 2018

Description
When using pip install --user inside a venv (created using the venv module coming with python), packages get installed in my ~/.local folder instead of the venv.
My pip.conf usually sets this flag per default, since I don't install packages globally. This bug makes that config option unusable.

Expected behavior
I would expect venv/bin/pip list to include the ipdb package. Instead, the package is installed outside of the venv.

How to Reproduce

python -m venv venv
venv/bin/pip install --user ipdb
venv/bin/pip list

Output

venv/bin/pip install --user ipdb
Collecting ipdb
  Using cached https://files.pythonhosted.org/packages/80/fe/4564de08f174f3846364b3add8426d14cebee228f741c27e702b2877e85b/ipdb-0.11.tar.gz
Requirement already satisfied: setuptools in ./venv/lib/python3.7/site-packages (from ipdb) (39.0.1)
Collecting ipython>=5.0.0 (from ipdb)
  Using cached https://files.pythonhosted.org/packages/f7/62/2fef7db3a7b75e8099c3d9db2630ae5ba0b9eefefd91f7497862393d90e8/ipython-6.5.0-py3-none-any.whl
Collecting pickleshare (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/9f/17/daa142fc9be6b76f26f24eeeb9a138940671490b91cb5587393f297c8317/pickleshare-0.7.4-py2.py3-none-any.whl
Collecting traitlets>=4.2 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl
Collecting backcall (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/84/71/c8ca4f5bb1e08401b916c68003acf0a0655df935d74d93bf3f3364b310e0/backcall-0.1.0.tar.gz
Collecting pexpect; sys_platform != "win32" (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl
Collecting simplegeneric>0.8 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip
Collecting jedi>=0.10 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/3d/68/8bbf0ef969095a13ba0d4c77c1945bd86e9811960d052510551d29a2f23b/jedi-0.12.1-py2.py3-none-any.whl
Collecting decorator (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl
Collecting pygments (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl
Collecting prompt-toolkit<2.0.0,>=1.0.15 (from ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/04/d1/c6616dd03701e7e2073f06d5c3b41b012256e42b72561f16a7bd86dd7b43/prompt_toolkit-1.0.15-py3-none-any.whl
Collecting ipython-genutils (from traitlets>=4.2->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl
Collecting six (from traitlets>=4.2->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting ptyprocess>=0.5 (from pexpect; sys_platform != "win32"->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl
Collecting parso>=0.3.0 (from jedi>=0.10->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/09/51/9c48a46334be50c13d25a3afe55fa05c445699304c5ad32619de953a2305/parso-0.3.1-py2.py3-none-any.whl
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.15->ipython>=5.0.0->ipdb)
  Using cached https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Installing collected packages: pickleshare, ipython-genutils, six, decorator, traitlets, backcall, ptyprocess, pexpect, simplegeneric, parso, jedi, pygments, wcwidth, prompt-toolkit, ipython, ipdb
  Running setup.py install for backcall ... done
  Running setup.py install for simplegeneric ... done
  Running setup.py install for ipdb ... done
Successfully installed backcall-0.1.0 decorator-4.3.0 ipdb-0.11 ipython-6.5.0 ipython-genutils-0.2.0 jedi-0.12.1 parso-0.3.1 pexpect-4.6.0 pickleshare-0.7.4 prompt-toolkit-1.0.15 ptyprocess-0.6.0 pygments-2.2.0 simplegeneric-0.8.1 six-1.11.0 traitlets-4.3.2 wcwidth-0.1.7

venv/bin/pip list
Package    Version
---------- -------
pip        18.0   
setuptools 39.0.1 

[bulletmark]

Geepers, I didn't know what the heck was going on until I just found this bug! Seems a serious bug. Also on Arch with Python 3.7.

[pradyunsg]

Given the confirmation from another user and the obvious severity of the bug, I've marked this as a blocker for 18.1; even though it's ~2 months away. :P

[bulletmark]

BTW, the bug happens with both pip versions 10.0.1 and 18.0 for me.

[pradyunsg]

Great; thanks for checking that @bulletmark! :)

[lorencarvalho]

@raffomania @bulletmark am I understanding correctly that you expect pip to ignore the --user flag if it's being ran inside a virtualenv? Just want to clarify.

[pradyunsg]

pip should refuse to install:

[lorencarvalho]

@pradyunsg yeah that's what I would expect

[piotr-dobrogost]

Erroring out is the plan described in issue #4575 "Adopting "working" scheme for every run":

Only packages in the same scheme as the working scheme can be modified. By modifying, I mean installing or uninstalling a package. Trying to modify a package in a different scheme is not allowed and pip would print a message and error out.

[bulletmark]

The issue of this bug is not to do with explicit specification of --user on the command line. The issue is that a user having user mode set in his ~/.config/pip/pip.conf currently can NEVER use pip inside a virtualenv, because it just ignores the virtualenv and installs to ~/.local. That is why I described the bug above as "serious". It seems to me that pip should ignore the user mode setting in ~/.config/pip/pip.conf when it is run inside a virtualenv.

Whether pip errors out on an explicit --user when run within a virtualenv, or just installs to ~/.local as the user has explicitly asked, is another matter and another debate. Personally I think pip in that case should respect the added --user.

It is arguably a design error that by default pip installs globally (see the long running and still open debate in issue #1668) so many users just set the user default in their config and forget about it, like I did a month or so ago then scratched my head for ages why my next virtualenv was not getting set up correctly.

[piotr-dobrogost]

This is duplicate of issue #4141 (Behavior of --user in virtualenv), which mentions error message shown also by @pradyunsg in his comment. It's strange OP did not get this error.

@bulletmark

Different behavior based on where an option was specified (configuration file vs command line) would be very odd.

[pfmoore]

@piotr-dobrogost Agreed. I'd be against having behaviour differ based on where the user specifies --user. What about an environment variable? Should a persistent environment variable be treated differently than PIP_USER=yes pip install foo? I don't think we want to start down this route...

I understand that people see setting --user in a config file as being somehow different in intention, as more of a choice to default to user than a request to always force user, but that's just not what it is.

many users just set the user default in their config

And that's the problem - setting user=true in the pip config file is not "setting the user default", rather it's "asking pip to use --user for all installs", which is subtly (but significantly) different. I guess there may be some value in adding a warning note to the docs explaining that point.

Having user installs be the default is what #1668 is about, and discussions about that should take place over there.

[bulletmark]

@pfmoore, so you are happy with the current situation that if you set user in your config you can NEVER pip install in a virtualenv?! That is the important problem here. What happens if you specify --user on the command line when running in a virtualenv is another less important issue.