Default to --user #1668

Open
dstufft opened this Issue Mar 21, 2014 · 112 comments

Projects

None yet
@dstufft
Member
dstufft commented Mar 21, 2014

When pip is installed on a system that has an OS Python install there is currently a problem where pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python. This creates an issue where people are using pip to manage system level packages when they should likely be using the system package manager.

So my intention is that pip should default to --user however there are a few sticking points with this:

  • How does this interact with Windows? Does this make sense there?
  • How does this interact with altinstall'd Pythons? Specially such as are installed with tools like https://github.com/yyuu/pyenv
  • What do we do for when people do invoke pip as root? Installing into /root/.local/ does not seem very useful.
  • What does this mean for get-pip and the pypa install instructions?
  • ~/.local/bin is not on many people's $PATH, is there anything that can be done about this?
  • --user installs lack precedence to global easy_install'd packages, which can be quite unexpected.

There are a number of issues that are relevant here: #624 #1443 #1153 #1500 #1051

/cc @ncoghlan

@qwcode
Contributor
qwcode commented Mar 21, 2014

What does this mean for the pypa install instructions?

currently the mention of root or Administrator access is in a footnote, so the instructions would fundamentally stay the same (assuming get-pip automatically defaulted to --user as well). the change would be to explain that they will end up with installs that only apply to their current user.

regardless of the --user change, the current instructions need to explain that some distributions like debian, are disabling ensurepip, so they shouldn't go looking for it as a way to get pip.

@dstufft
Member
dstufft commented Mar 21, 2014

Long term I'm pretty sure they are going to keep it enabled, they are just figuring out how to do it.

@qwcode
Contributor
qwcode commented Mar 21, 2014

although this issue is described as being about where pip should manage packages by default, almost more important (to me at least) is that this is indirectly about where get-pip will place pip by default.

@qwcode
Contributor
qwcode commented Mar 21, 2014

How does this interact with altinstall'd Pythons?
Specially such as are installed with tools like https://github.com/yyuu/pyenv

the user scheme is .local/lib/pythonX.Y/site-packages, so if you're managing multiple pythons with the same major/minor version, you could end up with a mess I guess.

for tools like that, that manage real pythons, pinning pip to global mode would make sense for those.

@dstufft
Member
dstufft commented Mar 21, 2014

A relevant issue on the redhat/fedora bug tracker https://bugzilla.redhat.com/show_bug.cgi?id=662034#c10

@dstufft
Member
dstufft commented Mar 21, 2014

It looks like for Fedora/RedHat PATH=$PATH:$HOME/.local/bin:$HOME/bin is already in their /etc/skel/.bash_profile. This would make stuff installed with --user still show up by default, at least for bash users (default shell). Maybe other distros already have this?

@pfmoore
Member
pfmoore commented Mar 21, 2014

There's nothing specially different about Windows that I know of. But we've only just got the standard Python directory onto people's PATH, I don't expect that getting a per-user scripts directory on is going to be particularly easy - and there may be technical difficulties as well, I'm not sure putting a user-level environment variable like %LOCALAPPDATA% into the system-level %PATH% even works :-(

I'm against rushing this. I would prefer waiting till we've had more experience with user installs and have managed to iron out the issues first. I know that's a chicken and egg problem, but I don't see rushing into a change as helping.

Also, with my Windows hat on, I'd have to say that making it harder for Unix users who haven't yet worked out that careless use of sudo is bad to do stupid things, isn't really that good a justification...

I'm assuming that this would only apply when running pip from a system Python.

@dstufft
Member
dstufft commented Mar 21, 2014

Well the different thing on Windows is there isn't a system provided Python like there is on *nix that also comes with system provided Python packages :)

I'm not particularly wanting to rush this either. I just wanted to start the discussion.

This doesn't really affect people who type sudo pip install as I think installing something as root should still go to the system site packages by default. This would be to guide people towards using --user when running as an unprivileged user. Right now what you get is that pip install as a regular user just bombs out with a permission error. Even if we improve that message to bomb out and suggest --user that's still not the greatest UX.

In talking to one of the Fedora people, I think the way it may make to do this is to implement a permissions check. If I have permission to write to the site-packages directory then I am a privledged user and pip installs to the global Python, If I do not then I am an unprivileged user and it installs to --user.

@ncoghlan
Contributor

As far as Windows goes, we usually duck the problem because we install Python to a non-privileged directory by default, so pip doesn't trigger UAC when installing globally. If someone changes that to install into Program Files instead, then UAC will trigger when they run pip (which is also OK). I'm not sure what happens if they select a per-user install in the installer.

As Paul notes, PATH on Windows doesn't include the user scripts directory yet (just the global one), so that's definitely worth taking into account.

I don't see any major barriers on the POSIX side though. How does this UX idea sound: for wheel based installs, check for permissions on the installation target directory, and if the user doesn't have write permissions, put up a prompt asking if they would like to do a --user install instead? A new "--global" or "--system" flag would force the "no" answer.

And then at some point in the future, we could skip the prompt and simply assume --user if the permissions weren't right for a global install.

@dstufft
Member
dstufft commented Mar 21, 2014

A prompt (except in the case of --no-input) probably makes sense for the transition step. I think checking permissions should make things work for Windows too since by default the location is in a spot where people have permissions on and so the only time they'd hit this would be if they are using a systems admin provided Python that explicitly didn't give them permission.

Essentially this makes our failure mode much better.

@qwcode
Contributor
qwcode commented Mar 21, 2014

making it harder for Unix users who haven't yet worked out that careless use of sudo
is bad to do stupid things, isn't really that good a justification..

sudo and security is really a secondary issue IMO.
this is primarily about preventing the conflict between OS packaging and pip in the system python.
Currently, linux users are often placed in an impossible situation.

  1. OS Mandate: 'Use the OS pkg mgr; Don't infect your system with roque pip-installed packages (including a get-pip'd pip)'
  2. Reality: "The OS packages (including pip) are too old, and I can't upgrade to the experimental release of the distro, just to get upgrades of some package. I'm going rogue...."
@qwcode
Contributor
qwcode commented Mar 21, 2014

another baby step is to document (assuming we test it) that you can get-pip.py --user, and have the PUG mention --user as a possibility for the install of virtualenv (and wheel, and twine too, if not in a virtualenv)

@ncoghlan
Contributor

FWIW, distro vendors also consider the status quo a problem and are looking at various ways to improve the tools for doing selective upgrades without impacting core OS components. Still worth us tackling from the upstream side though, since those efforts are at various stages of maturity and we want a consistent cross-platform solution for end users.

@qwcode qwcode referenced this issue in pypa/python-packaging-user-guide Mar 21, 2014
Open

document `--user` option for core tool installs #43

@pfmoore
Member
pfmoore commented Mar 22, 2014

I have no problem with changing things to help co-operate with distros on Unix; if defaulting to --user on Unix and not on Windows is acceptable, that's fine with me. I just don't see switching from a bad experience on Unix to a bad experience on Windows as a good trade-off. And I'm yet to be convinced that --user on Windows is ready for prime time.

Note that triggering UAC when running pip is bad on Windows, because what it actually means is that pip won't run except in an elevated console window. It does not mean that pip users get a nice prompt which they can say "OK" to, unfortunately, that's only how GUI apps work...

@dstufft
Member
dstufft commented Mar 22, 2014

@pfmoore What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

@dstufft
Member
dstufft commented Mar 22, 2014

FWIW I'm totally OK with making this optional depending on Windows or not. I'm just wondering if the check of "Do I have permissions to write to the site-packages folder" check won't achieve that anyways in 99% of installs, and if installing to --user would be a better experience in the fraction of cases where you don't have write permissions to the site-packages directory.

In other words, if we check permissions the proposal isn't replacing global install with a user install, it's replacing a permission error with a --user install.

@qwcode
Contributor
qwcode commented Mar 22, 2014

the proposal isn't replacing global install with a user install
it's replacing a permission error with a --user install

ok, but recall that the permissions check idea (#1048) won't the be the easiest thing

@dstufft
Member
dstufft commented Mar 22, 2014

It'll be pretty easy in the Wheel case, and we can get the 99.9% case covered with sdists, it should only be a problem in setup.py's that do really funky things.

@pfmoore
Member
pfmoore commented Mar 22, 2014

What do you think the user experience of Windows should be when you pip install something and you don't have permissions to write to the directory?

Well, being able to write to the site-packages directory is so rare on Windows that I'm not sure how relevant the question is in practice. But if it happened, I would say that an error saying "system site-packages is not writeable - did you mean to use --user?" would be the sensible approach.

Actually, I think that would be better on Unix as well. Switching the behaviour depending on writeability is bizarre, and it's not clear to me if using --user for a personally-built Python in a writeable directory is the right approach (but I have precisely zero experience of such a setup, so I don't have anything to back that opinion up).

Suggestion: Start with an error suggesting --user as above. Once people are using --user more commonly, and we have more experience with it, then let's consider switching the default.

@ncoghlan
Contributor

The approach Paul suggests is roughly what I had in mind, but with a yes/no prompt rather than bailing out with an error message. There are pros & cons to those two approaches (mostly relating to how they fail when invoked from another script).

@rkuska
rkuska commented Mar 24, 2014

Would it be possible to add to Python something like sys.localprefix (I see python-dev here)?
Sys.localprefix would be default to sys.prefix. Distros could define it same way as they define prefix (e.g. prefix = '/usr', localprefix='/usr/local'). Pip and easy_install would use sys.localprefix as a location to install to if used as sudo, if they don't have sufficient rights fall back to --user.

@bkabrda
bkabrda commented Mar 24, 2014

We've done something like this with "gem install" in Fedora 17 and Ruby devels were generally very satisfied about it, so I thought I'd share:

  • "gem install foo" installs "foo" into ~/somedir if uid != 0
  • "gem install foo" installs "foo" into /usr/local/somedir if uid == 0
  • "yum install rubygem-foo" installs RPM-packaged "foo" into /usr/somedir

I think that choosing install dir based on user privileges would be very confusing for people; "pip install foo" should work the same way for all non-root users. If you consider majority of users, they'll want to "pip install" into their home and "sudo pip install" into a system-wide dir. Therefore I think it's the best approach to do this based on uid as shown above - and for those users who are not root but want to install into a system-wide dir, there is always the --target option.

@Ivoz
Member
Ivoz commented Mar 24, 2014

@bkabrda that's an easy choice to make as long as you're not worried about adding scripts to $PATH. What have you done there?

@bkabrda
bkabrda commented Mar 24, 2014

@Ivoz as noted by @dstufft, Fedora already has $HOME/.local/bin on PATH, so everything worked out of the box (speaking of scripts).

@ncoghlan
Contributor

Slavek's suggestion sounds good to me for POSIX systems. It should also work nicely with containers, since stuff running in a container can be told to think it is uid 0, even though it's something else entirely from outside the container.

For Windows, I'm less sure what the right answer is. We don't have quite the same problem with getting into an argument with the system package manager, although it does exist to some degree (pip install vs MSI installers), and have the additional complication that even in Python 3.4, the installer's optional PATH modifications only add the global Scripts directory, not the per-user one.

Windows also has weirdness based on whether Python is installed to the default location (uncontrolled) or into Program Files (UAC privilege escalation needed to make changes).

@felixonmars

The "localprefix" idea suggested by @rkuska looks nice to me, simple and easy. Also I really like the approach @bkabrda suggests, especially for distros like Arch that make python 3.x the default python.

One more note is, Arch has disabled ensurepip in the 3.4.0 release just like Debian (mostly because we want to provide latest versions of setuptools and pip, while the bundled versions may be outdated for ages), so it would be nice to have a tip for it.

@lvoz
For Arch we didn't add any path under $HOME to PATH, but I still think it fine. We already have wiki entry for Ruby that tells the user to add it.

@ncoghlan
Contributor

It would be nice if rather than just disabling it, the Arch and Debian developers would help Slavek work on his patch to have ensurepip reconstruct a wheel from the system versions and install that into virtual environments.

@bkabrda
bkabrda commented Mar 24, 2014

Just FYI, we already have working Python 3.4 RPM builds in Fedora's Copr build system (testing, not advisable to install them if you don't want to break anything). If you want to know more about how we approach this, have a look at the code etc, see my discussion with Barry Warsaw [2] at debian-python ML.
(Our patches still need some love before we propose them upstream, but everything works ok for us downstream ATM)

[1] http://copr-fe.cloud.fedoraproject.org/coprs/bkabrda/python-3.4/
[2] https://lists.debian.org/debian-python/2014/03/msg00046.html

@felixonmars

Sorry, but AFAIK Arch never supported wheel, nor do we want to make ensurepip invoke package manager (correct me if I understand this approach wrong, though).

For ensurepip, it would be a nice idea to warn the user not to install pip using ensurepip though, and that's the most I can think of for now.

@bkabrda
bkabrda commented Mar 24, 2014

Our ensurepip will never invoke package manager. In short, Fedora's ensurepip will rely on system packaged setuptools and pip to be always present (our python3 package will depend on them, which will create a dependency loop, but we can handle that) and when user will create new virtualenv, it will repack system setuptools and pip into wheels and install them into the new virtualenv.
We're starting to get a bit off-topic, so I guess we should continue this discussion someplace else...

@dstufft
Member
dstufft commented Mar 24, 2014

@felixonmars Does that mean on Arch venv is broken the same as it is on Debian?

@felixonmars

OK, thanks for the explanation, I think I got the idea. I'll see what I can do and follow up on debian-python for this.

@dstufft Yes, the bundled version got installed instead of the system version.

@dstufft
Member
dstufft commented Mar 24, 2014

@bkabrda So that solves the problem if you're invoking pip with a system Python, we have to special case virtualenv/venv in that case (which isn't the end of the world). But it also means that we're assuming all Pythons are system Pythons, even ones installed by the user into their homedir such as with https://github.com/yyuu/pyenv (which is how I run Python actually). So I'm not sure, that's why I thought of the permission check.

@qwcode
Contributor
qwcode commented Mar 24, 2014

ditto what @dstufft said, in using a tool like pyenv to juggle non-root pythons, it would be pretty odd for root to be the key to do a global install (where "global" != "system")

@qwcode
Contributor
qwcode commented Mar 24, 2014

We've done something like this with "gem install" in Fedora 17

you mean the rpm of gem for fedora 17 has this as a distro hack?, or gem itself has this logic?

@qwcode
Contributor
qwcode commented Mar 24, 2014

one thought is that pip should provide a "system friendly mode" that can be turned on by distros that are packaging pip, or when users know they are installing pip themselves into a system-managed python. otherwise, pip does it's normal global-by-default logic.

@dstufft
Member
dstufft commented Mar 24, 2014

Ah @felixonmars I just saw https://projects.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/python , if all you're doing is calling --without-ensurepip that's fine. That leaves ensurepip module intact for the venv module. The patches Ubuntu (Debian?) currently has in place actually calls rm -rf on the ensurepip package itself, making it unavailable to be called within a venv.

@merwok
merwok commented Mar 24, 2014

I think defaulting to user installs was proposed by Nick two years ago, or maybe someone else longer ago, but the thread went into many directions and was not fruitful. Main obstacle was backward compat IIRC, but these days sysadmins and pip users may be more used to be careful with virtualenv/pip/setuptools updates, for better or worse. I don’t remember if that discussion happened on distutils-sig or pypa-dev.

@felixonmars

@dstufft Oh, I see... But I still feel wrong to have a bundled (maybe old) version installed while there are packages installed system-wise :)

@dstufft
Member
dstufft commented Mar 25, 2014

@felixonmars well the problem is you can't install a system package into a virtualenv afaik :) (And even if you could there are different concerns with what you want in a system package vs inside a virtualenv)

@felixonmars

@dstufft That's where Slavek's rewheel kicks in, but that's a bit off-topic :)

@felixonmars

@dstufft Thanks!

Not sure if I should continue the discussion there, because that shouldn't be a problem since we didn't unbundle pip in Arch, and rewheel should generate usable wheel :)

@bkabrda
bkabrda commented Mar 25, 2014

@dstufft you're right that we would probably need to specialcase virtualenv and possibly even pyenv, which seems ugly. I have to admit I'm not that familiar with pyenv, so I can't think of any "proper" solution for that right now. I'll try to have a closer look at it and think of something.

@qwcode for "gem install" on Fedora, it's a downstream patch. TBH I didn't touch Fedora's Rubygems for long, but the last time I checked, it was downstream only.

@rkuska
rkuska commented Mar 25, 2014

I have posted link to this issue to related bug in python [1]. I would like to move this discussion to python core, to have a configurable location for local installs of not only pip but also easy_install and python setup.py install, after that pip could use (e.g.) sys.localprefix dir if uid==0 and install to user dir if uid!=0 as @bkabrda mentioned.

For the time being I am fine with pip installing to user dir by default.

[1] http://bugs.python.org/issue1298835

@rkuska
rkuska commented Apr 1, 2014

As Nick pointed out I moved the discussion from issue1298835 to pypa-dev [1].
[1] https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0

@dstufft dstufft referenced this issue Apr 18, 2014
Open

Obsolete pip2014.com #1731

4 of 5 tasks complete
@dstufft dstufft added this to the Improve our User Experience milestone Apr 19, 2014
@hickford

Default to --user

Good idea!

pip install foo will throw an error because it doesn't have file permissions. This causes people to instead run sudo pip install foo which globally installs to the system Python

It's worse than that. Many people (such as users of a university computer network) don't have sudo privileges. When pip fails with a permission error, they will either:

  1. Email their administrator and ask them to install the package. That's slow and tedious for everyone.
  2. Give up and do without the package.

Only if they are particularly experienced or well-informed will they try pip install --user.

If we end up deciding against 'default to --user', we should consider the less disruptive alternatives:

  1. Fallback to --user (if install fails due to perimissions errors)
  2. (If install fails due to perimissions errors), encourage the user (in large friendly letters) to try --user
@Juanlu001

I actually have been using --user in all my installations for a while now, but generally people are not aware of this option. I agree it's annoying to see the permissions error every single time you forget the option. I noticed user installations are not compatible with conda though (conda/conda#448).

@zooba
zooba commented Feb 9, 2015

I'd like to revive this, as people are going to run into it on Windows far more often now that 3.5 defaults to installing into Program Files (which requires administrator privileges to modify).

I like the "fallback to --user on permission error" option best for installs, with "default to --user and fallback to system" on uninstalls and maybe add a --system or --system-only option to prevent the fallback. Users deliberately running pip with admin permissions will get a system install as expected, and currently users without permissions would get an error, so I don't believe there's any back-compat issues here.

My aim is for an uninformed user to be able to do pip install spam; python -c "import spam" without errors (though possibly a warning when retrying with --user).

@dstufft
Member
dstufft commented Feb 10, 2015

I've been thinking about this lately especially since #2403 and #2401 were opened recently that made me think about the use of sudo pip install and how that can often break systems outside of just the permissions problems.

To this end I wonder if the better end goal isn't just that --user is the default, no magical permissions checks, and add a new flag like --global, --system, or --no-user which will restore the previous behavior. This is easier to explain to people since it'll have the same behavior no matter how/where things are installed.

Now, if we decide that's the way we want to go, there is still the problem of how do we get to that point from where we are today. Obviously at whatever point we switch the default is going to be a fairly big breaking change so we're going to need a fairly long lead time. I think that if we decide to go that way the best way to do it would be to continue to keep the --user option around, add a new option like --global, and then if one or the other option isn't configured do the permission check magic that we've talked about and if the magic has decided to use the --global behavior we'll raise a very loud deprecation warning. This deprecation warning would stick around for a lot longer than our other deprecations that we've issued since this would be a fairly major change in behavior. I think that we would also want to add an option which would turn off the magic fallback and implement the future behavior so that people can get that behavior today.

Another important question is what should we do if we detect that the Python we're installing into does not have user sites enabled. In this case I would simply state that --user is an error and for those Pythons we'll simply fallback to defaulting to --global. This would cover the virtualenv/venv I believe automatically, and if someone wants a "heavier" weight isolated environment using fully compiled Pythons they can simply patch site.py so that they set ENABLE_USER_SITE to False at the top of the file.

@warsaw
warsaw commented Feb 10, 2015

+1, and I'd vote for --system.

I'll note, and I'm sure you'll love this, the Ubuntu default has changed:

https://launchpad.net/ubuntu/+source/python-pip/1.5.6-4ubuntu1

On Feb 09, 2015, at 04:41 PM, Donald Stufft wrote:

To this end I wonder if the better end goal isn't just that --user is the
default, no magical permissions checks, and add a new flag like --global,
--system, or --no-user which will restore the previous behavior. This
is easier to explain to people since it'll have the same behavior no matter
how/where things are installed.

@dstufft
Member
dstufft commented Feb 10, 2015

I think that --system is wrong because that really only applies to some Pythons. It doesn't apply to:

  • Python on Windows
  • pyenv installed Pythons
  • Conda installed Pythons
  • Nix Installed Pythons
  • Custom Compiled Pythons.

It only applies on *nix systems where the Python happens to be shipped by the system. I feel like --global is a much better name for that flag.

And FML, why does Debian/Ubuntu feel the need to constantly add random patches that break expected use cases?

@Tinche
Tinche commented Feb 10, 2015

So if the Ubuntu patch gets reverted, what's the timeline on this getting fixed here? A week/a month/3 months/6 months/a year...?

@dstufft
Member
dstufft commented Feb 10, 2015

It depends on what you mean by fixed.

Assuming for a moment we're using the idea I had 2 posts ago (which still needs sign off from the other pip developers) we could likely pretty quickly get the transition plan where the following holds true:

  • When --user is passed we always install into the user site packages.
  • When --global (or whatever) is passed we always install into the global site packages.
  • When neither is passed we implement a fallback method which:
    • Detects if we're in a virtual environment and if so acts as if --global was passed.
    • Detects if user site-packages is disabled and if so acts as if --global was passed.
    • Detects if the effective user does not have write permissions to sys.path, and if they do not act as if --user was passed.
    • Finally, if all other detection methods failed, act as if --global was passed and print a deprecation warning saying that at some point in the future the behavior of this will change and it will instead install into --user.

This won't stop someone from doing sudo pip install foo into their system Python, however it will print a warning that at some point this is going to mean --user and tell them to explicitly start using --global. Once the above has been released for a long enough time we could then remove the deprecated code path and simplify this so that it's simply:

  • When --user is passed we always install into the user site packages.
  • When --global (or whatever) is passed we always install into the global site packages.
  • When neither option is passed in we implement a fallback method which:
    • Detects if we're in a virtual environment and if so acts as if --global was passed.
    • Detects if user site-packages is disabled and if so acts as if --global was passed.
    • Finally uses --user as the default.

However, moving from the first behavior to the second behavior is going to have a fairly long timeline. That is going to be a really major breaking change which is going to break things for any software (such as salt, chef, puppet) or deployment script (like custom one off Fabric scripts baked into thousands of projects across the world) which expects that sudo pip install <foo> is going to install into the global Python. We'll need to give a long lead time for people to see the warning, and modify their software and scripts to pass an explicit --global flag if they actually need that behavior.

@ncoghlan
Contributor

I think Donald's suggested plan is a good one, although if the pip team decides to adopt that approach, we should also file a 3.5 release blocker issue with CPython to get the per-user scripts directory added to the path on Windows along with the global one. If 3.5 adds that along with the switch to installing into Program Files by default, then we should get the desired behaviour of installing without admin permissions continuing to "just work".

Being able to change the default install location via the pip config files may also be useful, allowing folks to opt in to "per-user by default" on their own systems, even if the default hasn't changed yet. (e.g. "default-install=user" vs "default-install=global")

@didrocks

@dstufft: I'm happy to give a hand to implement your proposal as told on the ubuntu bug if you need me. Once we get that behavior in trunk, I'll backport and replace the current ubuntu patch with this of course.

Keep me posted if you need any help.

@pfmoore
Member
pfmoore commented Feb 10, 2015

I'm +1 on @dstufft's plan. I'd go for --global as the name, for the reasons stated. We've been sitting on the fence long enough now, let's go for it.

O'n and +1 on @ncoghlan's suggestion that Python 3.5 add the user site directory to PATH on Windows. Let's avoid any further stumbling blocks for this transition.

@bkabrda
bkabrda commented Feb 10, 2015

I'm also +1 for @dstufft's plan and +1 for --global, although I'm still wondering about the sudo pip install foo needing --global. Is there in fact any need for root to have user site packages? Could we special case root to have site packages off by default? (Which would make sudo pip install foo work just as it does now)

I have never seen anyone using --user for root with the current state, but if someone actually knows such a use case, I'd like to know.

@pfmoore
Member
pfmoore commented Feb 10, 2015

One thought that occurred to me is how will the user site directory work with multiple versions of Python installed?

If I do pip install pytest --user in Python 2.7 and 3.4 (with Python 3.4 my default Python), which version will the py.test command execute? I have a suspicion that the two installs will overwrite each other so it's a "last one wins" situation, which is not good. Also, if it is "last one wins", and I install the 2.7 version after the 3.4 one, how would I fix that to make 3.4 the default again?

And what about uninstalls? Consider the following scenario (untested, because I'd need to set up a clean machine if I didn't want to risk breaking my main laptop):

    C:\Apps\Python34\python.exe -m pip install pytest --user
    C:\Apps\Python27\python.exe -m pip install pytest --user
    C:\Apps\Python34\python.exe -m pip uninstall -y pytest

First, why didn't the second command issue a warning that it was overwriting an existing file? Or would it? Whether it did or not, do I have any options on what to do?

Second, would the uninstall not fail because the checksum of py.test.exe wouldn't match the value in the RECORD file? If it does fail, how would I uninstall? If it doesn't fail, will it break my Python 2.7 copy of pytest (by deleting the exe)?

And finally (for now!) would the user scripts directory go before or after the system scripts directory in PATH? IMO, it should go after, as otherwise a user install of a package in a Python that was not requested by the user to be "the default Python" could override the same package installed in the "default" Python's system site-packages.

Hmm, maybe there are some open questions that need resolving before we make this change...

@ncoghlan
Contributor

I'd expect the name clashes in the shared Windows user script directory to be handled the same way they're handled on POSIX systems with their shared bin directories: Python 3 wouldn't install the unqualified script name for installed packages, so the unqualified name would refer to the Python 2 version.

Same goes for whether the system install or the per-user install takes precedence by default: put the user script directory after the system one.

@pfmoore
Member
pfmoore commented Feb 10, 2015

"Python 3 wouldn't install the unqualified script name for installed packages" - presumably you mean that pip and setuptools (under Python 3) wouldn't, here? So that's a change to those tools?

Oh, and bleh. My muscle memory is 100% trained to type "pip install foo" to install into my default Python (3.4). That's going to be hell to retrain.

Also, Windows users get to choose whether "python" means Python 2 or 3 (via "make this the default Python" and "Add to PATH") - unlike Unix users where the system Python makes the rules. So if I say I want to make Python 3.5 my default, why shouldn't "pip" (or any other unqualified name) refer to that version? It seems bizarre that a user with only Python 3.5 installed on Windows can't use unqualified names simply because of a Unix issue around how the system Python needs things named. (Apologies if my description of the Unix constraints is inaccurate - I don't really understand the issue there). People installing multiple Python versions need a solution, but that solution shouldn't affect the majority who only need one version.

Maybe the simplest answer is that the user scripts directory should be versioned, just like the user site packages directory is?

@pfmoore
Member
pfmoore commented Feb 10, 2015

Oh, and should this debate be on python-dev rather than here, as it's more generally about the user site directory?

@ncoghlan
Contributor

Reopening the PEP 370 design discussion for the Windows per-user scripts directory on python-dev would certainly be reasonable. It's odd that global script installations on Windows are scoped by Python version, but per-user scripts installations are not.

That contrasts with the situation on POSIX, which consistently uses a shared namespace for executables at both levels (global or per-user), so name conflicts are more likely to show up at installation time, rather than through name shadowing at runtime.

@ncoghlan
Contributor

As far as the general solution to handling parallel installation goes, that would be the "-m" switch - that way you always know which Python you're invoking, rather than guessing about the contents of a separate script's shebang line.

@pfmoore
Member
pfmoore commented Feb 10, 2015

Understood re -m, but last time that was brought up in the context of how to document things, the strong preference was that we should treat pip install foo as the canonical way to install things (with -m and versioned commands being treated as the exceptions for specialist cases.

The pip manual doesn't even mention alternatives (except in the one case of using -m to upgrade pip itself on Windows).

@ncoghlan
Contributor
@warsaw
warsaw commented Feb 10, 2015

On Feb 09, 2015, at 07:58 PM, ncoghlan wrote:

Being able to change the default install location via the pip config files
may also be useful, allowing folks to opt in to "per-user by default" on
their own systems, even if the default hasn't changed
yet. (e.g. "default-install=user" vs "default-install=global")

I like this. One of the things we're struggling with is how to resolve
upstream's more conservative approach to migrating to a --user default, with
distros competing goals of making pip system-safe and consistent with other
tools. E.g.

https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1419695
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=725848

If we had a system-wide configuration file to select the default, then at
least all the machinery involved would be identical, and the distros wouldn't
have to carry ugly deltas. All we would have to do would be to change the
config file and document that to our users. Problems that caused would be
ours to handle, but at least it would be easy to explain.

I don't think Debuntu currently installs a site-wide pip.conf file (I'll have
to double check).

https://pip.pypa.io/en/latest/user_guide.html#configuration

One complication: we install different pip scripts for Python 2 and Python 3,
so I think we'd need separate config files for the different versions of
Python, e.g. /etc/{xdg/pip/}/pip{2,3}.conf or some such.

@warsaw
warsaw commented Feb 10, 2015

On Feb 10, 2015, at 01:24 AM, Paul Moore wrote:

One thought that occurred to me is how will the user site directory work with
multiple versions of Python installed?

On Debian/Ubuntu (maybe all *nixes?) we have no collision because the
libraries get installed into ~/.local/lib/pythonX.Y/site-packages

We do get collisions on ~/.local/bin though. Installing a package with a
script 'foo' using pip install --user foo and pip3 install --user foo will
end up clobbering the ~/.local/bin/foo script.

@dstufft
Member
dstufft commented Feb 10, 2015

One downside of this change is that I think Fedora is the only downstream distro that adds ~/.local/bin to $PATH by default (and it should come before /usr/bin and /usr/local/bin just like the user site packages comes before site-packages). Ideally downstreams will be able to modify their systems so that ~/.local/bin is on $PATH by default.

@warsaw
warsaw commented Feb 10, 2015

On Feb 10, 2015, at 07:24 AM, Donald Stufft wrote:

One downside of this change is that I think Fedora is the only downstream
distro that adds ~/.local/bin to $PATH by default (and it should come
before /usr/bin and /usr/local/bin just like the user site packages comes
before site-packages). Ideally downstreams will be able to modify their
systems so that ~/.local/bin is on $PATH by default.

Yep, but we can make a separate decision about that. (Or rather, downstreams
distros can do that.)

@pfmoore
Member
pfmoore commented Feb 10, 2015

(and it should come before /usr/bin and /usr/local/bin just like the user
site packages comes before site-packages)

Hmm, @ncoghlan just said the opposite, that (on Windows, in that particular
context) the user scripts directory should be after the system one. Which
I agree with as long as the user scripts directory is unversioned. If it's
versioned, I'm less concerned. But that probably reflects the fact that
Windows users have never needed to endure unversioned scriipts clashing and
monstrosities like pip2/pip3, the way Unix users have ;-)

@dstufft
Member
dstufft commented Feb 10, 2015

Oh, and the script clobbering thing is a more general problem that isn't really related to --user exactly. The same problem exists everywhere except Windows --global installs. I think the solution to that problem is a secondary concern/problem that we should solve separately to this issue.

So current thoughts:

  • I think there is agreement on the general idea I posted earlier, so I think we'll want to move forward with something like that.
  • We're going to use the --global flag.
  • We likely want some sort of warning if we're installing into --user and ~/.local/bin isn't on the $PATH.
  • We'll want some sort of default-install option that'll essentially disable the fallback behavior and just hardcode to either --global or --user.
  • As of pip 6.0 we have machine specific config files located in /etc/ however these are not specific to the version of Python that is executing pip. I think that's a good feature though somewhat separate as well (and might go hand in hand with solving the script clobbering thing as well).

I see one major open question still: Long term what do we want to happen when someone does sudo pip install foo or rather, if someone has permission to write to the site-packages directory?

The options I can think of are:

  1. Install into --global. This is the most backwards compatible option and likely represents what users expect when they type that. However it has the downside that it'll (silently) mess with the global Python, which on many systems can cause broken systems.
  2. Install into --user, this will protect from broken systems (I think?) but I don't think anyone would expect the root user to get a /root/.local/ install.
  3. Just error out and require the user to select either --user or --global manually.
  4. Pick one of the first two options, but provide a switch that distros can turn in in /etc/ that will enable the third option.

The fourth option might be the best one but I feel like we should try to select one of the other options first if we can come to some sort of solution that works the same on all platforms/scenarios and makes sense on all of them as I would prefer to have consistent behavior cross platform if possible.

I feel like the third option is likely a bad one, since "I have write permission" is going to be the scenario that anyone on Windows is on pre Python 3.5, or anyone who uses Conda, or anyone who uses pyenv. Erroring out seems like the wrong thing to do in all of those cases and is soemwhat user unfriendly to boot.

So I guess between 1 and 2 it really comes down to how bad of a practice do we think it is for people to (conceptually) do sudo pip install --global foo. On Windows, Conda, pyenv, etc I feel like the answer to that is "we don't think it's a bad practice at all". On *nix I feel like maybe the answer is "users should be allowed to do what they want to their system but we should provide rails to lead them away from "bad" things".

So thinking about it more, I think I'd have to go with the first option being the correct option for us to go with. That solves the backwards compatibility issues more or less since the behavior we'd really be changing is that if you type pip install foo without permission to install into that global directories you'd install into --user instead of raising an error. I think that it's unlikely that anyone is out there really depending on pip raising an error in that case. I also think that pip install --user as the root user is unlikely to be what anyone expects or want and it's really just making a footgun for users.

So then my updated proposal would be:

  • When --user is passed we always install into the user site packages.
  • When --global is passed we always install into the global site packages.
  • When neither is passed we implement a fallback method which:
    1. Detects if we're in a virtual environment and if so acts as if --global was passed.
    2. Detects if user site-packages is disabled and if so acts as if --global was passed.
    3. Looks at default-install and if that exists uses --user or -global based on it.
    4. Detects if the effective user does not have write permissions to global site packages, and if they do not act as if --user was passed.
    5. Finally, if all other detection methods failed, act as if --global was passed.

This would mean no deprecation period and that the expectation is that in the absence of --user or --global we'll try to make the best guess based on the capabilities of the Python we're running in, the effective user, and any configured default install method.

@dstufft
Member
dstufft commented Feb 10, 2015

(and it should come before /usr/bin and /usr/local/bin just like the user
site packages comes before site-packages)

Hmm, @ncoghlan just said the opposite, that (on Windows, in that particular
context) the user scripts directory should be after the system one. Which
I agree with as long as the user scripts directory is unversioned. If it's
versioned, I'm less concerned. But that probably reflects the fact that
Windows users have never needed to endure unversioned scriipts clashing and
monstrosities like pip2/pip3, the way Unix users have ;-)

I'm not even thinking in terms of the script names clashing. I just think it's flat out wrong for our $PATH variable to act differently than sys.path does. As far as I know sys.path's ordering is: stdlib > user site packages > global site packages. I think it would really confusing if some-script's ordering was global bin dir > user bin dir while python -m some-script was user site packages > global site packages.

If there's a problem with the script directories (and versioned vs unversioned) then I think we should fix that problem but that it's not super relevant to what order they should be in, because I feel like anything other than matching what sys.path does is super wrong.

@pfmoore
Member
pfmoore commented Feb 10, 2015

Good point, we should assume (and recommend) that $PATH follows sys.path, with (in Windows terms) C:\PythonXY before %APPDATA%\PythonXY\Scripts before C:\PythonXY\Scripts. I'll note that on the python-dev thread.

@zooba
zooba commented Feb 10, 2015

Donald's latest proposal sounds good to me (same as my original one plus all the important details I left out :) ). I agree 100% with the "So thinking about it more..." paragraph, too.

As for configuring PATH on Windows, that's going to be nearly impossible. A system-wide installer can only configure system-wide environment variables and they can't include expansions, so you can't set different paths for each user. (The per-user installer can do it, but in that case --global will succeed and so there's no need.)

Also, because of Windows's broken PATH handling, system-wide settings always beat per-user settings. This lead to my half-suggestion a while back (on python-dev, IIRC) about using the py.exe launcher for unversioned scripts, so that pip.exe would always go to the latest system or user Python rather than the one that installed it, and (e.g.) pip.exe -3.5 would allow specific version selection.

The only way to make this really work is for the Python installer to start installing batch files to configure PATH on request (and maybe a shortcut that runs the batch file). So the first thing users would type is activate-py35 and then their PATH is configured correctly for 3.5. I'm very torn on this though, as I don't want to get into the vcvarsall.bat situation, but at the same time it'll be great for scripts that currently make assumptions about install locations or try and go through the registry to find it.

@zooba
zooba commented Feb 10, 2015

and maybe a shortcut that runs the batch file

In my mind, this is a "Python 3.5 (32-bit) Command Prompt" shortcut. Visual Studio users should recognize the parallel.

@dstufft
Member
dstufft commented Feb 10, 2015

I don't know enough about Windows to fully understand the implications of all of that, but I feel pretty bad about reverting to a situation where pip install foo which installs a script foo can't immediately be executed as foo on the command line. If we're installing to --user and that isn't on their path by default then that feels like a pretty major regression to me.

I'm not sure I understand exactly how using py.exe for the unversioned scripts would work in practice and whether or not it would actually solve the problem. I'm not personally tied to creating scripts exactly as we are today so if it does work I think we can maybe do it but I'd have to better understand the implications before I could give my +1 on it.

@pfmoore
Member
pfmoore commented Feb 10, 2015

As for configuring PATH on Windows, that's going to be nearly impossible. A system-wide installer can only configure system-wide environment variables and they can't include expansions, so you can't set different paths for each user.

Oh, yuck. I'd forgotten that.

and maybe a shortcut that runs the batch file
In my mind, this is a "Python 3.5 (32-bit) Command Prompt" shortcut. Visual Studio users should recognize the parallel.

... which is horrible for those of us who default to Powershell for everything.

but I feel pretty bad about reverting to a situation where pip install foo which installs a script foo can't immediately be executed as foo on the command line.

Agreed. This would be a bad regression, and needs to be addressed. I just wish someone could think of a way :-)

I'm not sure I understand exactly how using py.exe for the unversioned scripts would work in practice

I think the point (which is only somewhat related to py.exe) is that the exe wrapper, rather than invoking a specific, hard-coded Python interpreter as it does at present, should dynamically choose the interpreter based on "what is the default" somehow (registry, the py.exe ini file, whatever Python %PATH% gives you, ...). I'm not sure how it would help, though, as we'd still need a versioned user-scripts directory (to avoid files overwriting each other or files being run with interpreters that don't have the package installed) and we still wouldn't be able to put that at the right point in %PATH%.

@zooba
zooba commented Feb 10, 2015

... which is horrible for those of us who default to Powershell for everything.

I just tried it, and I can easily make activate-py35.bat and activate-py35.ps1 files that look and behave identically (i.e., everyone just gets to write activate-py35 and get the paths updated). Still not ideal, but possibly the best of a bad situation.

I'm not sure I understand exactly how using py.exe for the unversioned scripts would work in practice

I think the point (which is only somewhat related to py.exe) is that the exe wrapper, rather than invoking a specific, hard-coded Python interpreter as it does at present, should dynamically choose the interpreter based on "what is the default" somehow

The "somehow" is where py.exe comes in - the same rules should be used. If the launcher were updated to check its own name, then instead of launching python.exe it could try and launch 'scripts\\{}{}.{}.exe'.format(argv[0], major_version, minor_version) (with suitable extension trimming on argv[0], etc.) instead. Then installers need to use a different file for the unversioned launcher from the fully versioned one (which would be unchanged from today), but that file would just be the normal py.exe with a new name. (This could have been even simpler in the old days of pip-script.py files, but now that those are gone...)

But as you say, PATH is still the problem. I don't have any good solutions, and not even that many bad solutions. Environment variables are really meant for administrators or users to configure, not installers.

@dstufft
Member
dstufft commented Feb 10, 2015

We can adjust how we install scripts to make a better user experience. It's all a balance, the combined .exe and .py file made things nicer for users since there's just the singular .exe file. However I feel like getting the best story for pip install foo && foo is more important than that particular thing, so if we need to make the .exe and the .py separate again I think that's a good tradeoff for a good answer.

@zooba
zooba commented Feb 10, 2015

The .exe/.py separation is the least of the problems :)

I'm actually warming up to (or grudgingly accepting?) the activate-py35 idea. I've never been a fan of the installer adding Python to PATH at all, and the py -m pip command hasn't really gained traction (and nor would it work for, say, Sphinx). I'll write up a more detailed description of how activate-py could work and post it to python-dev to see if it gets any traction.

@dstufft
Member
dstufft commented Feb 10, 2015

Would these activate-whatever files be similar to the ones inside of virtual environment? Except instead of a virtual environment getting added to $PATH it's adding the real Pythons?

@zooba
zooba commented Feb 10, 2015

Yep, pretty much identical. On python-dev I've just suggested that they'd take a -x.y parameter, pass it to py.exe and use sysconfig to get the paths.

@dstufft
Member
dstufft commented Feb 10, 2015

Ok. I don't have an opinion on how bad that would be for Windows users since I'm not one. It doesn't sound absolutely horrible (maybe?) but that's about as far as my Windows knowledge can take me.

@warsaw
warsaw commented Feb 10, 2015

On Feb 10, 2015, at 08:02 AM, Donald Stufft wrote:

Oh, and the script clobbering thing is a more general problem that isn't
really related to --user exactly. The same problem exists everywhere
except Windows --global installs. I think the solution to that problem is
a secondary concern/problem that we should solve separately to this issue.

Agreed.

We're going to use the --global flag.

+1

We likely want some sort of warning if we're installing into --user and
~/.local/bin isn't on the $PATH.

+1, perhaps with a config option to suppress the warning?

We'll want some sort of default-install option that'll essentially
disable the fallback behavior and just hardcode to either --global or
--user.

+1

As of pip 6.0 we have machine specific config files located in /etc/
however these are not specific to the version of Python that is executing
pip. I think that's a good feature though somewhat separate as well (and
might go hand in hand with solving the script clobbering thing as well).

I would suggest a search order such as:

  • pipX.Y.conf
  • pipX.conf
  • pip.conf

rooted first(?) in /etc/xdg/pip then /etc, where X.Y of course is the
major.minor version number. First one found, wins.

I see one major open question still: Long term what do we want to happen when
someone does sudo pip install foo or rather, if someone has permission to
write to the site-packages directory?

Please note that there isn't just one "site-packages" directory. For example,
on Debian/Ubuntu we don't want sudo pip install to ever install into
/usr/lib, but only /usr/local/lib. And don't forget, it's dist-packages here
not site-packages (except in ~/.local because $reasons).

This is really an important point that sometimes gets forgotten (even by me).
Donald describes it best when he says there is a site-local directory and a
vendor-local directory. On Debian, site-local is
/usr/lib/pythonX.Y/dist-packages and no pip command should ever touch that,
while vendor-local is /usr/local/lib/pythonX.Y/dist-packages and it is a
documented and popular use case for some system administrators to pip install
into that directory.

So what I would suggest, since we're already down the config file path, is to
make these configurable, e.g.

  • global_install_directory
  • global_install_as_sudo (true/false)
  1. Install into --global. This is the most backwards compatible option
    and likely represents what users expect when they type that. However it has
    the downside that it'll (silently) mess with the global Python, which on many
    systems can cause broken systems.

With the above configurability, this would be totally compatible with Debian.
Debian superusers expect sudo pip install to go into
/usr/local/lib/pythonX.Y/dist-packages. I wouldn't say it "messes with the
system" because it doesn't break the package manager, and while it can
override system installed packages (which live in
/usr/lib/pythonX.Y/dist-packages), it does so in a defined, documented way.

  1. Install into --user, this will protect from broken systems (I think?)
    but I don't think anyone would expect the root user to get a
    /root/.local/ install.

Agreed, that would be unexpected.

  1. Just error out and require the user to select either --user or
    --global manually.

I'd be okay with that, but see the configuration switch above.

  1. Pick one of the first two options, but provide a switch that distros can
    turn in in /etc/ that will enable the third option.

Hey, what a great option!

The fourth option might be the best one but I feel like we should try to
select one of the other options first if we can come to some sort of solution
that works the same on all platforms/scenarios and makes sense on all of them
as I would prefer to have consistent behavior cross platform if possible.

I'm okay with having different, platform-defined behavior, if say, pip had a
--dry-run option that would at least tell the user exactly what it was going
to do and where.

I feel like the third option is likely a bad one, since "I have write
permission" is going to be the scenario that anyone on Windows is on pre
Python 3.5, or anyone who uses Conda, or anyone who uses pyenv. Erroring out
seems like the wrong thing to do in all of those cases and is soemwhat user
unfriendly to boot.

I'd be happy if the default configuration was to allow --global install if you
have permission. I don't even think Debian would change it (maybe others have
a different opinion, but at least that gives us options).

So I guess between 1 and 2 it really comes down to how bad of a practice do
we think it is for people to (conceptually) do sudo pip install --global foo.

On Debian, it is an expected use case, as long as it goes to /usr/local/lib.
It cannot go to /usr/lib.

(All this describes outside-a-venv-behavior BTW.)

  • When --user is passed we always install into the user site packages.

+1

  • When --global is passed we always install into the global site
  • packages.

Global vendor packages == +1, global site packages == -1.

  • When neither is passed we implement a fallback method which:
    1. Detects if we're in a virtual environment and if so acts as if
      --global was passed.

+1, although I'll repeat that this directory inside a venv is named
/lib/pythonX.Y/site-packages

  1. Detects if user site-packages is disabled and if so acts as if
    --global was passed.

Hmm, I'm not sure about this one.

  1. Looks at default-install and if that exists uses --user or
    -global based on it.

+1

  1. Detects if the effective user does not have write permissions to global
    site packages, and if they do not act as if --user was passed.

+1

  1. Finally, if all other detection methods failed, act as if --global
    was passed.

Also not sure.

This would mean no deprecation period and that the expectation is that in the
absence of --user or --global we'll try to make the best guess based
on the capabilities of the Python we're running in, the effective user, and
any configured default install method.

Sounds like a reasonable goal.

@dstufft
Member
dstufft commented Feb 10, 2015

To be clear, when I talk about "global site packages", I primarily mean "whatever distutils tells us to install things to globally". Since Python itself doesn't have a "vendor local" vs "site local" distinction this is going to be the same directory on any downstream that doesn't patch their Python. On debian however it's going to actually be a dist-packages directory and "global" actually means "site local". That's not something that pip needs to do anything special to support though because that is taken care of by Debuntu's patches to Python.

In other words, pip already installs to /usr/local/../dist-packages/ on Debuntu since Debuntu has patched distutils to support that distinction. The only time pip would mess with /usr/../dist-packages/without the user passing in a flag of some kind is that pip would uninstall files from there (because pip doesn't see that directory as special, it just sees it as a thing on sys.path). Debian has patched pip to prevent that uninstall from there though, and I think real support for that is dependent on getting real support for "vendor local" site packages in Python upstream.

@pfmoore
Member
pfmoore commented Feb 10, 2015

We likely want some sort of warning if we're installing into --user and ~/.local/bin isn't on the $PATH

How fragile is this warning likely to be? Would it convert to absolute path? Would it be case insensitive on case insensitive filesystems? Would it treat \ and / as equivalent on Windows? What about symlinks?

If you're installing a package that doesn't install any scripts, the warning's irrelevant anyway.

Personally, I think that a warning will probably do more harm than good.

@dstufft
Member
dstufft commented Feb 10, 2015

We likely want some sort of warning if we're installing into --user and
~/.local/bin isn't on the $PATH.

+1, perhaps with a config option to suppress the warning?

We could just make it use a Python warnings thing so people could just silence
it using the built in Python warnings. If we feel the need to silence it at
all.

As of pip 6.0 we have machine specific config files located in /etc/
however these are not specific to the version of Python that is executing
pip. I think that's a good feature though somewhat separate as well (and
might go hand in hand with solving the script clobbering thing as well).

I would suggest a search order such as:

  • pipX.Y.conf
  • pipX.conf
  • pip.conf

rooted first(?) in /etc/xdg/pip then /etc, where X.Y of course is the
major.minor version number. First one found, wins.

Split this out into #2417 as I believe it's only tangently related to this
discussion.

I see one major open question still: Long term what do we want to happen when
someone does sudo pip install foo or rather, if someone has permission to
write to the site-packages directory?

Please note that there isn't just one "site-packages" directory. For example,
on Debian/Ubuntu we don't want sudo pip install to ever install into
/usr/lib, but only /usr/local/lib. And don't forget, it's dist-packages here
not site-packages (except in ~/.local because $reasons).

This is really an important point that sometimes gets forgotten (even by me).
Donald describes it best when he says there is a site-local directory and a
vendor-local directory. On Debian, site-local is
/usr/lib/pythonX.Y/dist-packages and no pip command should ever touch that,
while vendor-local is /usr/local/lib/pythonX.Y/dist-packages and it is a
documented and popular use case for some system administrators to pip install
into that directory.

So what I would suggest, since we're already down the config file path, is to
make these configurable, e.g.

  • global_install_directory
  • global_install_as_sudo (true/false)

pip doesn't really control these (I mean, obviously at the high level it does
because it's the one moving files around), they come from distutils/sysconfig.

I think that pip should probably never touch a vendor-local directory unless
some sort of flag was given like --vendor which would enable vendors to use
pip in their build chain. This concept doesn't really exist outside of debuntu
right now and their patches already handle this, so I don't think it's super
relevant to this discussion except to note that when I say
"global site packages" I meant /usr/local/.../dist-packages on Debubuntu
and the "site-local" packages at some point if Python accepts the Debuntu
system upstream.

  1. Install into --global. This is the most backwards compatible option
    and likely represents what users expect when they type that. However it has
    the downside that it'll (silently) mess with the global Python, which on many
    systems can cause broken systems.

With the above configurability, this would be totally compatible with Debian.
Debian superusers expect sudo pip install to go into
/usr/local/lib/pythonX.Y/dist-packages. I wouldn't say it "messes with the
system" because it doesn't break the package manager, and while it can
override system installed packages (which live in
/usr/lib/pythonX.Y/dist-packages), it does so in a defined, documented way.

Yea, "messes with system" may be too harsh. Mostly I mean that
sudo pip install foo may break things if the system depends on foo and
you install an incompatible version of things. However that same case holds
true for basically anything you install into`/usr/local``.

  • When --global is passed we always install into the global site
  • packages.

Global vendor packages == +1, global site packages == -1.

You mean this the other way around right?

  • When neither is passed we implement a fallback method which:
    1. Detects if we're in a virtual environment and if so acts as if
      --global was passed.

+1, although I'll repeat that this directory inside a venv is named
/lib/pythonX.Y/site-packages

Yea, again we just ask Python where that directory is, we don't compute it
ourselves, so "where that directory is" is a venv/virtualenv thing.

  1. Detects if user site-packages is disabled and if so acts as if
    --global was passed.

Hmm, I'm not sure about this one.

I don't think there's any way around this, if user site packages is disabled I
don't think that we should install into there because we have no way to know
if that disabling means there is no user site-packages, or why. We can only
assume that there is none.

  1. Finally, if all other detection methods failed, act as if --global
    was passed.

Also not sure.

This basically boils down to:

If we haven't found a reason we should use --user, like a --user flag, or
not having permissions to write to the directory, then don't use it. This
is what will keep sudo pip install foo without a --global flag
working. This also means this is the most importnat rule for not breaking
working software that expects sudo pip install foo to act this way.

@dstufft
Member
dstufft commented Feb 10, 2015

We likely want some sort of warning if we're installing into --user and ~/.local/bin isn't on the $PATH

How fragile is this warning likely to be? Would it convert to absolute path? Would it be case insensitive on case insensitive filesystems? Would it treat \ and / as equivalent on Windows? What about symlinks?

If you're installing a package that doesn't install any scripts, the warning's irrelevant anyway.

Personally, I think that a warning will probably do more harm than good.

Well we could implement it as the (cross platform equivilant of):

def user_bin_on_path():
    paths = os.environ.get("PATH").split(":")
    for path in paths:
        if os.path.realpath(path) == os.path.realpath(USER_BIN_DIR):
            return True
    return False

Something like that should handle symlinks, path seperators, and the like I think. Devils in the details and maybe we can't get it accurate enough to cover the various cross platform corner cases.

I do agree that the warning is irrelevant if there are no scripts being installed, and I meant to say that we'd only bother with the warning when we're installing scripts.

The basic idea is that I don't want someone to do something like pip install [--user] foo where the --user is implicit, and then do foo and get a "command not found" without any guidance about why it might not be found. A warning at least will tell them they need to add things to the PATH. It may also help prompt users on Windows to add their user dir to their PATH manually (or run some script to do it) which might alleviate the need for the activate scripts that Steve pointed out. Although I think that would still mean that System values take precedence over user values so that still might be confusing since the order of PATH and sys.path would be reversed still.

@pfmoore
Member
pfmoore commented Feb 10, 2015

Doh. I forgot that realpath canonicalised the path. Yes, of course that works well enough.

The thing that bothers me is that if the test mistakenly sees an issue, the user gets an annoying and incorrect warning with no way of avoiding it other than changing their system to work around a mistake on pip's part. But given that the test you're proposing is probably robust enough, and we only do the check if we're installing scripts, the issue is probably rare enough to be ignorable.

@dstufft
Member
dstufft commented Feb 10, 2015

Alright, I think we have some broad agreement here. I think the next step is to get a patch up. I'll go ahead and try to get something up later tonight or tomorrow.

@warsaw
warsaw commented Feb 10, 2015

On Feb 10, 2015, at 12:14 PM, Donald Stufft wrote:

The basic idea is that I don't want someone to do something like pip install [--user] foo where the --user is implicit, and then do foo and
get a "command not found" without any guidance about why it might not be
found.

Debian, and maybe other Linuxes, actually have a command-not-found package
which prompts the user to install a package if they invoke a command that
isn't installed.

% inkscape
The program 'inkscape' is currently not installed. You can install it by typing:
sudo apt-get install inkscape

Maybe we could hook into that for supporting platforms.

@warsaw
warsaw commented Feb 10, 2015

On Feb 10, 2015, at 12:06 PM, Donald Stufft wrote:

  • When --global is passed we always install into the global site
  • packages.

Global vendor packages == +1, global site packages == -1.

You mean this the other way around right?

I could have my terminology mixed up, so I'll be explicit:

/usr/local/lib/pythonX.Y/dist-packages == +1
/usr/lib/pythonX.Y/dist-packages == -1

but as you say previously, this isn't something that pip really needs to worry
about since it's inherited from distutils and Debuntu's distutils is patched
to do the right thing.

  1. Detects if user site-packages is disabled and if so acts as if
    --global was passed.

Hmm, I'm not sure about this one.

I don't think there's any way around this, if user site packages is disabled I
don't think that we should install into there because we have no way to know
if that disabling means there is no user site-packages, or why. We can only
assume that there is none.

Oh, I see what you mean now. I guess that makes sense. My concern is that it
may be more difficult to predict what pip is actually going to do, but as I
say, I'd be happy with a --dry-run flag so that pip can unequivocally tell
you what it's going to do.

  1. Finally, if all other detection methods failed, act as if --global
    was passed.

Also not sure.

This basically boils down to:

If we haven't found a reason we should use --user, like a --user flag, or
not having permissions to write to the directory, then don't use it. This
is what will keep sudo pip install foo without a --global flag
working. This also means this is the most importnat rule for not breaking
working software that expects sudo pip install foo to act this way.

I see. I think that's okay, as the main anti-use case we want to avoid is:

normal_user$ pip install foo
HEY YOU HAVE NO PERMISSIONS
normal_user$ sudo pip install foo
HEY YOU JUST BORKEDED YOUR SYSTEM

and the "not having permissions to write to the directory implies --user" rule
takes care of that.

@dstufft
Member
dstufft commented Feb 10, 2015

Oh yea I forgot about command-not-found. Could be useful. By the way do you know the right place to propose that devubtu has the user local bin on the path by default?

On Feb 10, 2015, at 4:05 PM, Barry Warsaw notifications@github.com wrote:

On Feb 10, 2015, at 12:14 PM, Donald Stufft wrote:

The basic idea is that I don't want someone to do something like pip install [--user] foo where the --user is implicit, and then do foo and
get a "command not found" without any guidance about why it might not be
found.

Debian, and maybe other Linuxes, actually have a command-not-found package
which prompts the user to install a package if they invoke a command that
isn't installed.

% inkscape
The program 'inkscape' is currently not installed. You can install it by typing:
sudo apt-get install inkscape

Maybe we could hook into that for supporting platforms.


Reply to this email directly or view it on GitHub.

@warsaw
warsaw commented Feb 10, 2015

On Feb 10, 2015, at 01:17 PM, Donald Stufft wrote:

Oh yea I forgot about command-not-found. Could be useful. By the way do you
know the right place to propose that devubtu has the user local bin on the
path by default?

Not definitively, but maybe the adduser or passwd packages? The former owns
/usr/sbin/adduser and the latter owns /usr/bin/useradd.

@dstufft
Member
dstufft commented Feb 10, 2015

First pass: #2418

@Mikaela
Mikaela commented May 26, 2015

I am not sure if this helps, but…

  • ~/.local/bin is not on many people's $PATH, is there anything that can be done about this?

Some other projects use /etc/profiled.d/ for adding directories to $PATH. At least on Antergos I see the following (unless I misunderstand what these do) jre.csh, jre.sh, perlbin.csh, perlbin.sh.

# Do not change this unless you want to completely by-pass Arch Linux' way
# of handling Java versions and vendors. Instead, please use script `archlinux-java`
setenv PATH "${PATH}:/usr/lib/jvm/default/bin"
# Do not change this unless you want to completely by-pass Arch Linux' way
# of handling Java versions and vendors. Instead, please use script `archlinux-java`
export PATH=${PATH}:/usr/lib/jvm/default/bin
# Set path to perl scriptdirs if they exist
# https://wiki.archlinux.org/index.php/Perl_Policy#Binaries_and_Scripts
# Added /usr/bin/*_perl dirs for scripts
# Remove /usr/lib/perl5/*_perl/bin in next release

[ -d /usr/bin/site_perl ] && setenv PATH ${PATH}:/usr/bin/site_perl
[ -d /usr/lib/perl5/site_perl/bin ] && setenv PATH ${PATH}:/usr/lib/perl5/site_perl/bin

[ -d /usr/bin/vendor_perl ] && setenv PATH ${PATH}:/usr/bin/vendor_perl
[ -d /usr/lib/perl5/vendor_perl/bin ] && setenv PATH ${PATH}:/usr/lib/perl5/vendor_perl/bin

[ -d /usr/bin/core_perl ] && setenv PATH ${PATH}:/usr/bin/core_perl

# If you have modules in non-standard directories you can add them here.
#export PERLLIB=dir1:dir2
# Set path to perl scriptdirs if they exist
# https://wiki.archlinux.org/index.php/Perl_Policy#Binaries_and_Scripts
# Added /usr/bin/*_perl dirs for scripts
# Remove /usr/lib/perl5/*_perl/bin in next release

[ -d /usr/bin/site_perl ] && PATH=$PATH:/usr/bin/site_perl
[ -d /usr/lib/perl5/site_perl/bin ] && PATH=$PATH:/usr/lib/perl5/site_perl/bin

[ -d /usr/bin/vendor_perl ] && PATH=$PATH:/usr/bin/vendor_perl
[ -d /usr/lib/perl5/vendor_perl/bin ] && PATH=$PATH:/usr/lib/perl5/vendor_perl/bin

[ -d /usr/bin/core_perl ] && PATH=$PATH:/usr/bin/core_perl

export PATH

# If you have modules in non-standard directories you can add them here.
#export PERLLIB=dir1:dir2
@nchammas

As a random pip user, I just want to say thank you for working on user experience issues like this and like the others captured in the Improve our User Experience milestone. Is there someplace I can contribute financially to this effort?

As an aside, I use Homebrew and always appreciated the fact that I could just brew install something and it would work without any fuss over permissions.

Once I understood why installing with Homebrew never required sudo, I tried to get into the habit of doing pip install --user something, but ironically Homebrew disables this due to a reported bug in distutils.

Given the popularity of Homebrew on OS X, perhaps y'all want to look into this purported incompatibility of Homebrew and pip install --user.

Anyway, +1 from me on making --user the default.

@warsaw
warsaw commented May 29, 2015

A quick follow up. Didier uploaded a change to Ubuntu's pip in vivid that set --user as default. I mostly also allowed it but now I think it was a mistake and I intend to revert that for wily (but probably won't SRU a change for vivid). See https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1460203 for details and rationale.

@ncoghlan
Contributor

+1 for letting upstream drive the timing of this change, as the user vs system distinction is already confusing enough without platform differences coming into play.

That said, it may be reasonable for the Linux distros to start flipping the default to --user as soon as --global support lands in upstream pip (pip 8?), since doing global installs on Linux, rather than per-user or per-venv installs, is well established as being a bad idea with potentially unpredictable side effects.

@dstufft
Member
dstufft commented May 30, 2015

I told @warsaw privately, but I'll say it here too:

I think this change is important and I plan on coming back around to my pull request to finish it up. However, I'm one person stretched over multiple projects and right now Warehouse is more important to me than this change is because PyPI legacy is becoming increasingly burdensome to operate and I feel we need to get that replaced as soon as possible. Therefore my time on pip isn't focusing on making many changes myself, rather it's focusing on guiding through changes other contributors make as well as any release process stuff.

That being said, if anyone really cares about this feature landing sooner rather than later they can pick up my work and finish it and I'll be happy to review (and hopefully merge) it. I'm happy to talk through that with anyone who wants to do it and provide guidance on the change. Otherwise it's going to wait until after I finish up the items that are more pressing to me.

@hickford

Perhaps worth repeating at this juncture:

If we end up deciding against 'default to --user', we should consider the less disruptive alternatives

  1. Fallback to --user (if install fails due to perimissions errors)
  2. (If install fails due to perimissions errors), encourage the user (in large friendly letters) to try --user
@zooba
zooba commented Jun 1, 2015

FWIW, now that the Windows installer for Python 3.5 strongly encourages a per-user installation, I'm not so concerned about this change occurring in pip. It would probably make a better configuration option for distros (with the --global override) than a new default or a fallback.

@hugorodgerbrown hugorodgerbrown added a commit to yunojuno/trifecta that referenced this issue Jun 8, 2015
@hugorodgerbrown hugorodgerbrown Fix permissions with pip installation
Using the --user option. See pypa/pip#1668 for
more details.
52343e2
@tbekolay

Just wanted to pop in to +1 this issue and say thanks for working on this! We're running a scientific Python summer school currently and have had many people sudo pip install when they ran into permissions issues.

FWIW I'm in favor of defaulting to --user, or falling back to --user if install fails due to permission errors. In this case, I would prefer to give them the directions to reverse what they've done (e.g., We installed to your user directory. If you don't want this, then 'pip uninstall my_package') rather than failing and telling them how to try again.

@rbtcollins
Contributor

I see a lot of bug reports on mock due to --user - at least on Ubuntu where --user's place on the path is after dist-packages, and so what pip installed isn't actually used, for any common dependency (like six) that is in dist-packages.

@dstufft
Member
dstufft commented Oct 12, 2015

They put it after dist-packages? That's a broken installation IMO.

@ncoghlan
Contributor

Right, user installs should be before site(/dist)-packages, and then system scripts & utilities should be running with -I (or -Es in Python 2).

@rbtcollins
Contributor

Yes, I agree :). It may be fixed in vivid, but I've definitely seem reports where things are coming from totally the wrong place.

I just used the system pip to install setuptools on my vivid machine, to experiment.

>>> import setuptools
>>> setuptools.__path__
['/home/robertc/.local/lib/python2.7/site-packages/setuptools']
>>> import sys
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/robertc/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/home/robertc/work/mock']

So this one is sane. But confusingly this was introduced in vivid - so IDK. I'll see if I can dig up a reproducer at some point (to file in the Ubuntu BTS)

@dstufft
Member
dstufft commented Oct 12, 2015

Paging @warsaw

@Lartza Lartza referenced this issue in KenT2/tboplayer Nov 24, 2015
Closed

sudo pip install should not be used #45

@takluyver
Contributor

User site-packages can end up getting rearranged to after global site-packages by setuptools' evil .pth files. These were a recurring problem for me until I learned never to use setup.py directly - I'd install something, and suddenly I was using older versions of all kinds of other things. After a few times, I wrote an aggressive tool to un-mangle my system.

@pradyunsg
Contributor

I think at the very least, for the time-being, pip should just print a friendly message asking the user to run --user if the command fails. While switching it to be the default has it's benefits, printing a better message shouldn't be too hard.

It's a nice "quick-fix" for the problem till we actually move to defaulting to --user.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment