Deprecate pip, pipX, and pipX.Y

[dstufft]

Currently, people are regularly running into problems over confusion about which particular Python a particular invocation of pip is going to manage. There are many cases where what Python a particular invocation of pip* is going to invoke is unclear:

• pip3 install --upgrade pip will overwrite pip and possibly switch it from pointing to 2.7 to 3.5.
• pip3 isn't specific enough, you might have 3.4 and 3.5 installed.
• pip3.4 isn't specific enough, you might have 3.4.0 and 3.4.1 installed.
• pip3.4.0 isn't specific enough, you might have multiple copies of 3.4.0 installed in various locations.
• We don't have a good answer for what the pip binary should be called on alternative Python interpreters like PyPy (pip-pypy? What if we have two versions of PyPy? pip-pypy2.6? What if we have PyPy and PyPy3? pip-pypy-2.6 and pip-pypy3-2.6?).

Overall, it's become increasingly clear to me that this is super confusing to people. Python has a built in mechanism for executing a module via python -m. I think we should switch to using this as our preferred method of invocation. This should completely eliminate the confusion that is caused when python and pip don't point to the same version of Python, as well as solve the problem of what do you call the binary on alternative implementations.

In addition to the confusion, we also have the fact that pip install --upgrade pip doesn't actually work on Windows because of problems with the .exe file being open. However python -m pip install --upgrade pip does work there.

I see only three real downsides:

• There is a lot of documentation, examples, code or other instances of inertia using just pip and this will be churn for those.
• It's 10 more letters to type.
• It doesn't work on Python 2.6.

For the first of these, I think the answer is to just have a very long deprecation cycle, in the order of years. I wouldn't even put a specific date on it's removal, I'd just add the warnings and re-evaluate in the future. Luckily we've shipped support for python -m pip for quite some time, so it won't be something that people need to deal with version differences (mostly).

The second of these I don't really have a great answer for, I think that 10 extra letters probably isn't that big of a cost to pay for the reduced confusion and the default answer working on Windows. We could possibly offer a recipe in the docs to restore pip, pipX, and pipX.Y via shell aliases.

This last item is the biggest sticking point for me. As far as I know, Python 2.6 still has far too many users for us to drop it since, as of 6 months ago, it was still ~10% of the traffic on PyPI (source). The problem with Python 2.6 is that it only supports -m when the target is a module not a package. I see four possible solutions to this:

• Don't deprecate pip* on Python 2.6.
• Add a module like pipcli.py that people can invoke like python -m pipcli instead of python -m pip.
• Move pip/ to _pip/ and make pip.py.
• Document that due to limitations of Python 2.6, it will need to be invoked as python -m pip.__main__.

I don't really like the pipcli idea, the other three all have pros and cons, but I think I personally live with either not deprecating pip* on Python 2.6 and/or documenting that it needs to be invoked as ``python -m pip.main on Python 2.6).

What do the @pypa/pip-developers think?

[RonnyPfannschmidt]

i like the move from pip to _pip,
that way pip's implementation goes to a more "private" namespace
the expense is breaking tools that go into pips internals

[RonnyPfannschmidt]

i like the move from pip to _pip,
that way pip's implementation goes to a more "private" namespace
the expense is breaking tools that go into pips internals

[dstufft]

The other expense is that if we ever want to make a public API we're either limited to having a single namespace (whatever is in pip.py) or we need to change it back to a package (and possibly break Python 2.6 again, unless we've deprecated it by then).

Of course, we may never make a public API in which case, the point is moot.

[dstufft]

The other expense is that if we ever want to make a public API we're either limited to having a single namespace (whatever is in pip.py) or we need to change it back to a package (and possibly break Python 2.6 again, unless we've deprecated it by then).

Of course, we may never make a public API in which case, the point is moot.

[dstufft]

I can't help but think that @warsaw and @ncoghlan probably have some opinions on this too.

[dstufft]

I can't help but think that @warsaw and @ncoghlan probably have some opinions on this too.

[dstufft]

Maybe @bkabrda too! and @tdsmith

[dstufft]

Maybe @bkabrda too! and @tdsmith

[Lukasa]

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here. Otherwise, I think this is a good idea: +1.

[Lukasa]

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here. Otherwise, I think this is a good idea: +1.

[dholth]

-1 on removing pip I would have to change all of my deployment scripts.
+1 on removing pipX and pipX.Y

On Tue, Oct 6, 2015, at 08:16 AM, Cory Benfield wrote:

Don't underestimate the power of that first point. The intertia is
high: lots of tools will assume pip, and lots of documentation
will be wrong. Having a long deprecation cycle is basically mandatory
here. Otherwise, I think this is a good idea: +1.

— Reply to this email directly or view it on GitHub[1].

Links:

1. #3164 (comment)

[dholth]

-1 on removing pip I would have to change all of my deployment scripts.
+1 on removing pipX and pipX.Y

On Tue, Oct 6, 2015, at 08:16 AM, Cory Benfield wrote:

Don't underestimate the power of that first point. The intertia is
high: lots of tools will assume pip, and lots of documentation
will be wrong. Having a long deprecation cycle is basically mandatory
here. Otherwise, I think this is a good idea: +1.

— Reply to this email directly or view it on GitHub[1].

Links:

1. #3164 (comment)

[dstufft]

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here.

Yea, completely agree. I essentially assume that we should not have a defined removal date (and possibly never) and just have it log a message to stderr.

[dstufft]

Don't underestimate the power of that first point. The intertia is high: lots of tools will assume pip, and lots of documentation will be wrong. Having a long deprecation cycle is basically mandatory here.

Yea, completely agree. I essentially assume that we should not have a defined removal date (and possibly never) and just have it log a message to stderr.

[Lukasa]

I essentially assume that we should not have a defined removal date (and possibly never) and just have it log a message to stderr.

I don't think that will work. Just printing annoying warnings with no defined "your shit will break no later than X" date doesn't really help: people will just ignore the warnings. I think if you want to do this you should decide a date (possibly one far away, but still). One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

[Lukasa]

I essentially assume that we should not have a defined removal date (and possibly never) and just have it log a message to stderr.

I don't think that will work. Just printing annoying warnings with no defined "your shit will break no later than X" date doesn't really help: people will just ignore the warnings. I think if you want to do this you should decide a date (possibly one far away, but still). One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

[dstufft]

-1 on removing pip I would have to change all of my deployment scripts.
+1 on removing pipX and pipX.Y

I don't think it makes sense to deprecate (not talking about removal any time soon) pipX and pipX.Y without also doing it for pip since that is arguably the worse offender of them all.

[dstufft]

-1 on removing pip I would have to change all of my deployment scripts.
+1 on removing pipX and pipX.Y

I don't think it makes sense to deprecate (not talking about removal any time soon) pipX and pipX.Y without also doing it for pip since that is arguably the worse offender of them all.

[piotr-dobrogost]

One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

This is good idea.

[piotr-dobrogost]

One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

This is good idea.

[dstufft]

One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

That's November 30, 2020 for the end of RHEL 6 Production 3 phase. They have a super special extended life cycle beyond that, but perhaps we could just target 2020 and if we roll around to 2020 and Python 2.6 is still somehow in wide support, we push it back further.

[dstufft]

One possible date to start the discussion: when the last RHEL LTS release with Python 2.6 stops being supported (that's very far away still, but worth discussing).

That's November 30, 2020 for the end of RHEL 6 Production 3 phase. They have a super special extended life cycle beyond that, but perhaps we could just target 2020 and if we roll around to 2020 and Python 2.6 is still somehow in wide support, we push it back further.

[dholth]

Another alternative would be to stop requiring that pip execute in the same Python environment that it is installing things into...

[dholth]

Another alternative would be to stop requiring that pip execute in the same Python environment that it is installing things into...

[dstufft]

To be honest, I'm not sure how pip install -p python2.7 is any better than python2.7 -m pip install. We have to inspect the Python we're installing into to get information from it, so either we're going to subshell into that Python to shuffle data back and forth (like my half done virtualenv rewrite does) or we'll need to continue to be executed by the same Python environment. Feels like shuffling deck chairs more than anything else.

[dstufft]

To be honest, I'm not sure how pip install -p python2.7 is any better than python2.7 -m pip install. We have to inspect the Python we're installing into to get information from it, so either we're going to subshell into that Python to shuffle data back and forth (like my half done virtualenv rewrite does) or we'll need to continue to be executed by the same Python environment. Feels like shuffling deck chairs more than anything else.

[dholth]

For that particular idea the main benefit would be that you would only have to upgrade pip once.

[dholth]

For that particular idea the main benefit would be that you would only have to upgrade pip once.

[dstufft]

That much is true, the flip side of that is it makes it (somewhat) harder to support versions of pip older than what pip itself supports, since the installs are no longer independent (or you'll need to keep around an older copy installed somewhere else). On the other hand, pip could more easily drop support for running the main pip binary command in a particular Python, while keeping compatibility for installing into that version of Python. It would (continue) to enable bundling all of pip into a single zip file that can be executed somewhat independently of actually having it installed.

It doesn't address the fact that pip install --upgrade pip on Windows blows up because the OS has a handle open to pip.exe though, which I don't think can be solved without using python -m, at least if I understand @pfmoore correctly.

[dstufft]

That much is true, the flip side of that is it makes it (somewhat) harder to support versions of pip older than what pip itself supports, since the installs are no longer independent (or you'll need to keep around an older copy installed somewhere else). On the other hand, pip could more easily drop support for running the main pip binary command in a particular Python, while keeping compatibility for installing into that version of Python. It would (continue) to enable bundling all of pip into a single zip file that can be executed somewhat independently of actually having it installed.

It doesn't address the fact that pip install --upgrade pip on Windows blows up because the OS has a handle open to pip.exe though, which I don't think can be solved without using python -m, at least if I understand @pfmoore correctly.

[bkabrda]

I think @sYnfo wants to comment on this more than I do, since I no longer maintain Python in Fedora/RHEL.

[bkabrda]

I think @sYnfo wants to comment on this more than I do, since I no longer maintain Python in Fedora/RHEL.

[dstufft]

Ah, that's right, I forgot. Sorry!

[dstufft]

Ah, that's right, I forgot. Sorry!

[bkabrda]

(But I personally think that even if these are removed, we'll still provide them on distribution level in form that is best for the set of Python interpreters that we ship; at least that was my first idea when I read the proposal... We have a general policy for Python executables that mandates this in Fedora.)

[bkabrda]

(But I personally think that even if these are removed, we'll still provide them on distribution level in form that is best for the set of Python interpreters that we ship; at least that was my first idea when I read the proposal... We have a general policy for Python executables that mandates this in Fedora.)

[pfmoore]

• +1 for deprecation - but I'd be happy enough simply to deprecate in documentation, not make the scripts themselves moan at the user.
• It's only 6 more letters on Windows - py -m pip :-)
• For 2.6, I'm OK with either letting people continue to use pip or advising python2.6 -m pip.__main__. I don't think it's worth making changes to pip to give them any other solution.

The inertia issue is huge, and I don't think we should fight it directly. Rather, we should switch the documentation to use the "python -m pip" form, and make that form official PyPA policy (by which I mean we take pains to use that form consistently in whatever we post, etc). Maybe offer PRs for the install documentation of well-known projects to switch them to the new form. We can worry about formally deprecating and/or removing the scripts once the python -m pip form starts to gain a bit of traction in common usage.

[pfmoore]

• +1 for deprecation - but I'd be happy enough simply to deprecate in documentation, not make the scripts themselves moan at the user.
• It's only 6 more letters on Windows - py -m pip :-)
• For 2.6, I'm OK with either letting people continue to use pip or advising python2.6 -m pip.__main__. I don't think it's worth making changes to pip to give them any other solution.

The inertia issue is huge, and I don't think we should fight it directly. Rather, we should switch the documentation to use the "python -m pip" form, and make that form official PyPA policy (by which I mean we take pains to use that form consistently in whatever we post, etc). Maybe offer PRs for the install documentation of well-known projects to switch them to the new form. We can worry about formally deprecating and/or removing the scripts once the python -m pip form starts to gain a bit of traction in common usage.

[tdsmith]

I think my success criterion is making this the incantation that people get from StackOverflow. Shoot for the moon, etc etc.

I think publishing a linkable intent-to-deprecate message with a rationale may help convince third-party maintainers to accept documentation PRs.

The messaging here is tricky, because deprecating pip doesn't mean deprecating pip...

[tdsmith]

I think my success criterion is making this the incantation that people get from StackOverflow. Shoot for the moon, etc etc.

I think publishing a linkable intent-to-deprecate message with a rationale may help convince third-party maintainers to accept documentation PRs.

The messaging here is tricky, because deprecating pip doesn't mean deprecating pip...

[dstufft]

I think the only real alternative is Daniel's suggestion. I think the current situation sucks and I can't really think of a way to save the attempt to manage pip versions using a suffix or prefix or anything that doesn't end up actually specifying which interpreter you want to run under.

[dstufft]

I think the only real alternative is Daniel's suggestion. I think the current situation sucks and I can't really think of a way to save the attempt to manage pip versions using a suffix or prefix or anything that doesn't end up actually specifying which interpreter you want to run under.

[sYnfo]

I'm +1 for deprecation—it seems to make a whole lot of sense from upstream point of view and I don't see any issue this could cause downstream.

In Fedora this would mean shipping all the binaries during the deprecation period, as we do now; and not shipping them at all afterward. Perfect sync with upstream, hopefully. :) I'll make sure all the Fedora docs get updated, when this goes official.

( @bkabrda If I understand the guidelines correctly, they only mandate shipping all the MAJOR.MINOR executables iff there are any executables in the first place, this issue seems to be about removing the pip executables entirely, right? )

Also +1 to either keeping pip or using python -m pip.main with python 2.6.

[sYnfo]

I'm +1 for deprecation—it seems to make a whole lot of sense from upstream point of view and I don't see any issue this could cause downstream.

In Fedora this would mean shipping all the binaries during the deprecation period, as we do now; and not shipping them at all afterward. Perfect sync with upstream, hopefully. :) I'll make sure all the Fedora docs get updated, when this goes official.

( @bkabrda If I understand the guidelines correctly, they only mandate shipping all the MAJOR.MINOR executables iff there are any executables in the first place, this issue seems to be about removing the pip executables entirely, right? )

Also +1 to either keeping pip or using python -m pip.main with python 2.6.

[erikrose]

A couple of points that haven't come up:

• Doesn't this just move the problem to python? Why is python (of various versions and in various virtualenvs) any easier to keep track of than pip?
• If you're using a virtualenv (which most people should be), pip unambiguously refers to the active one. An alternative solution would be to encourage people to always use venvs, which solves other problems as well, e.g. dependency conflicts from dropping everything into a global environment.

[erikrose]

A couple of points that haven't come up:

• Doesn't this just move the problem to python? Why is python (of various versions and in various virtualenvs) any easier to keep track of than pip?
• If you're using a virtualenv (which most people should be), pip unambiguously refers to the active one. An alternative solution would be to encourage people to always use venvs, which solves other problems as well, e.g. dependency conflicts from dropping everything into a global environment.

[rschoon]

I'm not big on the idea, but I don't think I've ever run pip outside of virtualenv, which is usually activated into my current shell, so the extra typing doesn't gain me anything (I'm selfish!). Of course, if it's not activated, I probably have to type a lot more than that anyway...

How would people feel about making an exception for virtualenvs? There's no mystery for which python it is using, but then again it might be too confusing to have it work differently between for virtualenv and the system install, so I'm feeling pretty humble about its quality as a suggestion. I'm also not sure there's a good solution for checking that.

Also,

pip3.4 isn't specific enough, you might have 3.4.0 and 3.4.1 installed.

Does this really come up often? I'm curious how you're distinguishing between them if so, since I didn't think python would normally install itself any more specific than by the pythonX.Y name (and unless they have different prefixes, sounds like it would also try to share site-packages anyway).

[rschoon]

I'm not big on the idea, but I don't think I've ever run pip outside of virtualenv, which is usually activated into my current shell, so the extra typing doesn't gain me anything (I'm selfish!). Of course, if it's not activated, I probably have to type a lot more than that anyway...

How would people feel about making an exception for virtualenvs? There's no mystery for which python it is using, but then again it might be too confusing to have it work differently between for virtualenv and the system install, so I'm feeling pretty humble about its quality as a suggestion. I'm also not sure there's a good solution for checking that.

Also,

pip3.4 isn't specific enough, you might have 3.4.0 and 3.4.1 installed.

Does this really come up often? I'm curious how you're distinguishing between them if so, since I didn't think python would normally install itself any more specific than by the pythonX.Y name (and unless they have different prefixes, sounds like it would also try to share site-packages anyway).

[rschoon]

Oh, I guess that does apply having multiple 3.4.0 installs too, but then it seems like you'd definitely be using full paths to distinguish whether you're using "pip" or "python -m pip", but since you're using full paths anyway, the argument against having extra wordiness goes away.

[rschoon]

Oh, I guess that does apply having multiple 3.4.0 installs too, but then it seems like you'd definitely be using full paths to distinguish whether you're using "pip" or "python -m pip", but since you're using full paths anyway, the argument against having extra wordiness goes away.

[mattrobenolt]

I may be a little disconnected here, but why not replace pip with an alias that was effectively something like:

alias pip=/usr/bin/env python -m pip

Or a script that was akin to this to be installed into /usr/local/bin/pip.

That way we don't lose pip, and it universally works errywhere.

Also not sure if that solves the problems you're facing, just my $0.02.

[mattrobenolt]

I may be a little disconnected here, but why not replace pip with an alias that was effectively something like:

alias pip=/usr/bin/env python -m pip

Or a script that was akin to this to be installed into /usr/local/bin/pip.

That way we don't lose pip, and it universally works errywhere.

Also not sure if that solves the problems you're facing, just my $0.02.

[dstufft]

@erikrose
It does sort of shift the problem to python, but typically the confusion that I've seen arise stems from when python and pip disagree. Like you might have /usr/bin/python and /usr/local/bin/python and if python points to /usr/local/bin/python and pip points to /usr/bin/python it's a recipe for confusion. So we completely eliminate the confusion cased by having the two disagree, but removing any ability for it to disagree. You're still left with confusion about what python means but I don't think there's anything at all that can be done about that, and particularly not by us.

A virtual environment makes the issue harder to hit, but I think that it's still possible. My gut tells me that new users (the ones most likely to get tripped up by this) are probably not going to be religiously using virtual environments (if they're using them at all).

@rschoon
We could possibly do that, it'd require a permanent special case inside of pip when pip installs itself (because there's no setup.py in a virtual environment) but it's certainly a possibility. My main fear with that is it feels like it'd just always be better to use python -m pip anyways because it works inside and outside of a virtual environment instead of having to remember to switch commands based on whether you're in one or not.

I'm not sure exactly how often the 3.4.0 and 3.4.3 thing comes up. I know that it's not super unusual on OSX since you have system provided Python sitting in /usr/bin/python and you might also install the Python.org installer or Homebrew or Macports or pyenv (or some combination of the above).

[dstufft]

@erikrose
It does sort of shift the problem to python, but typically the confusion that I've seen arise stems from when python and pip disagree. Like you might have /usr/bin/python and /usr/local/bin/python and if python points to /usr/local/bin/python and pip points to /usr/bin/python it's a recipe for confusion. So we completely eliminate the confusion cased by having the two disagree, but removing any ability for it to disagree. You're still left with confusion about what python means but I don't think there's anything at all that can be done about that, and particularly not by us.

A virtual environment makes the issue harder to hit, but I think that it's still possible. My gut tells me that new users (the ones most likely to get tripped up by this) are probably not going to be religiously using virtual environments (if they're using them at all).

@rschoon
We could possibly do that, it'd require a permanent special case inside of pip when pip installs itself (because there's no setup.py in a virtual environment) but it's certainly a possibility. My main fear with that is it feels like it'd just always be better to use python -m pip anyways because it works inside and outside of a virtual environment instead of having to remember to switch commands based on whether you're in one or not.

I'm not sure exactly how often the 3.4.0 and 3.4.3 thing comes up. I know that it's not super unusual on OSX since you have system provided Python sitting in /usr/bin/python and you might also install the Python.org installer or Homebrew or Macports or pyenv (or some combination of the above).

[dstufft]

@mattrobenolt

Basically, because it's super confusing if you're doing something like running myvenv/bin/pip without activating the virtual environment first or if you have a copy of Python installed to a non standard location /opt/my-app and you run /opt/my-app/bin/pip and expect to manage /opt/my-app/bin/python.

[dstufft]

@mattrobenolt

Basically, because it's super confusing if you're doing something like running myvenv/bin/pip without activating the virtual environment first or if you have a copy of Python installed to a non standard location /opt/my-app and you run /opt/my-app/bin/pip and expect to manage /opt/my-app/bin/python.

[qwcode]

it feels like a loss for virtual environments to lose a basic pip script when they don't suffer from this problem.

as for real pythons I guess I wouldn't mind seeing a more exact pip-<python binary path> type console script, and let the distro packagers tack on simpler scripts for system-managed pips.

[qwcode]

it feels like a loss for virtual environments to lose a basic pip script when they don't suffer from this problem.

as for real pythons I guess I wouldn't mind seeing a more exact pip-<python binary path> type console script, and let the distro packagers tack on simpler scripts for system-managed pips.

[Ivoz]

-1 on deprecating pip, +1 on the others.
I want to expand on some framing thoughts I have, that might help for discussing what is such a large issue. Skip to the break if you just want to read my thoughts on solutions.

For a start, you want to look at why we have this problem, and is it analogous to anyone else? It comes from having and allowing multiple pythons on the one running system. So system package managers do not have this problem, because for instance, you don't (can't) run debian jessie and debian wheezy live at the same time on one system; so it doesn't need to manage a libreoffice3.5 and libreoffice4.3 for example. However many other language package managers start having to deal with the same problem as pip's when they too have multiple versions installed. As @erikrose mentions, even python itself already runs into this issue on deciding on what python now is when more than one is installed.

I also want to look at the issue from the POV of majority python users. Note that this has become a real pain point mostly (or more-so) for people with more than 2 pythons installed. Otherwise pip would work, or simply pip2 and pip3 (and there probably wouldn't be enough inertia for everyone to start discussing). Beyond that it starts getting really complicated. But I'd believe most python users are happy using just one python. Even if their system somehow gives them 2 at some point, or they manage to install multiple, instead of upgrading / (removing previous and installing a new one) - if they got things right, they'd be fine with only one python going, and by extension, that python's pip. For all of these users suddenly taking away pip makes absolutely no sense and is just painful.

The other big source of pain is when someone merely tries to install a new python over a previous existing one, but the story for the entire environment being migrated to the new one (or "taking over the old one") isn't there. In that case I want to make the distinction that this is the install story's fault, not the existence of multiple pythons. For instance someone hoping to "install a new python!" but not getting it on their path. Even uninstalling the old python, might do nothing about giving them the environment they want (the new python on their path).

Also note that while the number of users collecting problems with managing pips may be small, their complaints are the only ones heard. I'd venture that their opinions are probably the majority on this discussion as well (because they're the ones with the issue). The silent majority doesn't care until things get changed for them, but we should still look to represent their use case fairly. Not of course, that those complaints therefore can't be valid.

---

Now with that in mind, here's solutions I like:

• clearly pip<versionstuff> doesn't scale well at all as a solution. So I'm in agreement for removing it. Most of the time its better to wait for a decent solution (if it exists) than try and keep going with one that creates as many issues as it solves.

• Stop replacing pip (or other pip2s or pip3s, even) without asking. Even system package managers do this. They ask beforehand. So should we. This way at the very least, the user sees straight away that there's an issue and perhaps can make a decision for themselves what they want pip to be. One can do a lot with this - look at who's python installation the existing pip comes from - its the same one as the current pip trying to install itself, then this could be fine. Otherwise make the user say --yes or --replace or answer Y to a prompt. Note that this could help with the same problem with other userland-programs-that-come-from-pypi. Make sure the user wants the executable script replaced. Even if this means we have to wait a long time to tell people that they might have to interact after calling pip install by default (for scripted uses of pip), and give them time to add --replace-scripts (or w/e) to their callouts, so be it.

• Start outputting some information about where things come from in an install! This could solve a lot of issues straight away. If I install a package with pip, pip doesn't speak about

  • what python it's on/from
  • what version it is, where its installed
  • where it is installing the new package(s)

  This would all be super useful information to know. It will immediately show me if the pip I'm calling on the command line is not the actual pip that I want, which is what tricks a LOT of users. It will also show me I'm installing for the right python and into the right site-packages.

I especially believe implementing the last two points above, would remove a lot of average user-problems in relation to this issue. In many cases they would be empowered to know the problem and solution themselves.

[Ivoz]

-1 on deprecating pip, +1 on the others.
I want to expand on some framing thoughts I have, that might help for discussing what is such a large issue. Skip to the break if you just want to read my thoughts on solutions.

For a start, you want to look at why we have this problem, and is it analogous to anyone else? It comes from having and allowing multiple pythons on the one running system. So system package managers do not have this problem, because for instance, you don't (can't) run debian jessie and debian wheezy live at the same time on one system; so it doesn't need to manage a libreoffice3.5 and libreoffice4.3 for example. However many other language package managers start having to deal with the same problem as pip's when they too have multiple versions installed. As @erikrose mentions, even python itself already runs into this issue on deciding on what python now is when more than one is installed.

I also want to look at the issue from the POV of majority python users. Note that this has become a real pain point mostly (or more-so) for people with more than 2 pythons installed. Otherwise pip would work, or simply pip2 and pip3 (and there probably wouldn't be enough inertia for everyone to start discussing). Beyond that it starts getting really complicated. But I'd believe most python users are happy using just one python. Even if their system somehow gives them 2 at some point, or they manage to install multiple, instead of upgrading / (removing previous and installing a new one) - if they got things right, they'd be fine with only one python going, and by extension, that python's pip. For all of these users suddenly taking away pip makes absolutely no sense and is just painful.

The other big source of pain is when someone merely tries to install a new python over a previous existing one, but the story for the entire environment being migrated to the new one (or "taking over the old one") isn't there. In that case I want to make the distinction that this is the install story's fault, not the existence of multiple pythons. For instance someone hoping to "install a new python!" but not getting it on their path. Even uninstalling the old python, might do nothing about giving them the environment they want (the new python on their path).

Also note that while the number of users collecting problems with managing pips may be small, their complaints are the only ones heard. I'd venture that their opinions are probably the majority on this discussion as well (because they're the ones with the issue). The silent majority doesn't care until things get changed for them, but we should still look to represent their use case fairly. Not of course, that those complaints therefore can't be valid.

---

Now with that in mind, here's solutions I like:

• clearly pip<versionstuff> doesn't scale well at all as a solution. So I'm in agreement for removing it. Most of the time its better to wait for a decent solution (if it exists) than try and keep going with one that creates as many issues as it solves.

• Stop replacing pip (or other pip2s or pip3s, even) without asking. Even system package managers do this. They ask beforehand. So should we. This way at the very least, the user sees straight away that there's an issue and perhaps can make a decision for themselves what they want pip to be. One can do a lot with this - look at who's python installation the existing pip comes from - its the same one as the current pip trying to install itself, then this could be fine. Otherwise make the user say --yes or --replace or answer Y to a prompt. Note that this could help with the same problem with other userland-programs-that-come-from-pypi. Make sure the user wants the executable script replaced. Even if this means we have to wait a long time to tell people that they might have to interact after calling pip install by default (for scripted uses of pip), and give them time to add --replace-scripts (or w/e) to their callouts, so be it.

• Start outputting some information about where things come from in an install! This could solve a lot of issues straight away. If I install a package with pip, pip doesn't speak about

  • what python it's on/from
  • what version it is, where its installed
  • where it is installing the new package(s)

  This would all be super useful information to know. It will immediately show me if the pip I'm calling on the command line is not the actual pip that I want, which is what tricks a LOT of users. It will also show me I'm installing for the right python and into the right site-packages.

I especially believe implementing the last two points above, would remove a lot of average user-problems in relation to this issue. In many cases they would be empowered to know the problem and solution themselves.

[pfmoore]

So, we're now in a situation where we don't support Python 2.6. So python -m pip works everywhere and I'd be in favour of using it as the canonical way of running pip in the docs (and yes, there's python2/python3 on Unix, and py on Windows, but we can probably rely on users to work that part out).

For the case of virtual environments, we could probably just install the pip executable when installing in a virtualenv (without --user or --target) and not otherwise, leaving it to the system packages (or ensurepip) to install a pip executable as needed in that situation. People who want to continue using pip can, as a convenience - we wouldn't drop that option, just stop mentioning it in the docs.

[pfmoore]

So, we're now in a situation where we don't support Python 2.6. So python -m pip works everywhere and I'd be in favour of using it as the canonical way of running pip in the docs (and yes, there's python2/python3 on Unix, and py on Windows, but we can probably rely on users to work that part out).

For the case of virtual environments, we could probably just install the pip executable when installing in a virtualenv (without --user or --target) and not otherwise, leaving it to the system packages (or ensurepip) to install a pip executable as needed in that situation. People who want to continue using pip can, as a convenience - we wouldn't drop that option, just stop mentioning it in the docs.

[dstufft]

One option could be to move the script wrapper to it's own package pip-cli or something like that, and just have that included in the list of packages that virtual environments install by default.

[dstufft]

One option could be to move the script wrapper to it's own package pip-cli or something like that, and just have that included in the list of packages that virtual environments install by default.

[pfmoore]

How would we handle cases like pip 10, where different versions of pip-cli are needed to support pip 9 and pip 10 (because pip.main moved)? If the user upgrades pip, we don't have a means to force upgrade pip-cli. So we'd end up with the same breakages that we currently see with OS wrappers.

The option of having pip-cli call pip in a subprocess is not reasonable, IMO. I know it's what we recommend for external tools, but I don't think that adding the overhead of an additional process on what will certainly remain the most common way of invoking pip for a long time, is really acceptable. You're effectively doubling the cost of Python startup, which is already high. I'd rather see some concrete measurements instead of gut feelings, but in the absence of measurements, I'm concerned about the cost of an extra subprocess.

If we do want to go down the pip-cli route, and we can't justify the subprocess approach, then either we need to give pip-cli a dispensation to use pip's internals, or we need to formally expose pip.main as a supported API (with whatever caveats we deem appropriate).

[pfmoore]

How would we handle cases like pip 10, where different versions of pip-cli are needed to support pip 9 and pip 10 (because pip.main moved)? If the user upgrades pip, we don't have a means to force upgrade pip-cli. So we'd end up with the same breakages that we currently see with OS wrappers.

The option of having pip-cli call pip in a subprocess is not reasonable, IMO. I know it's what we recommend for external tools, but I don't think that adding the overhead of an additional process on what will certainly remain the most common way of invoking pip for a long time, is really acceptable. You're effectively doubling the cost of Python startup, which is already high. I'd rather see some concrete measurements instead of gut feelings, but in the absence of measurements, I'm concerned about the cost of an extra subprocess.

If we do want to go down the pip-cli route, and we can't justify the subprocess approach, then either we need to give pip-cli a dispensation to use pip's internals, or we need to formally expose pip.main as a supported API (with whatever caveats we deem appropriate).

[pradyunsg]

Having pip-cli use pip.__main__.main should be fine.

[pradyunsg]

Having pip-cli use pip.__main__.main should be fine.

[dstufft]

Or just use runpy: https://docs.python.org/2/library/runpy.html, it's what -m uses anyways.

[dstufft]

Or just use runpy: https://docs.python.org/2/library/runpy.html, it's what -m uses anyways.

[pfmoore]

+1 on using runpy.

[pfmoore]

+1 on using runpy.

[pradyunsg]

+1 on using runpy.

Ditto. :)

TIL about runpy.

[pradyunsg]

+1 on using runpy.

Ditto. :)

TIL about runpy.

[Juanlu001]

You mean https://docs.python.org/3/library/runpy.html, right? 😏

[Juanlu001]

You mean https://docs.python.org/3/library/runpy.html, right? 😏

[pradyunsg]

@Juanlu001 we're still supporting Python 2.7. :)

[pradyunsg]

@Juanlu001 we're still supporting Python 2.7. :)

[pfmoore]

There's a bit more discussion over at #5508 (before @dstufft pointed out it was a duplicate of this issue!)

[pfmoore]

There's a bit more discussion over at #5508 (before @dstufft pointed out it was a duplicate of this issue!)

[pradyunsg]

Personally, I think a big source of confusion noted here is the mismatching pip vs python -m pip.

ISTM this is something that can be checked using sys.executable and PATH. Additionally, pip should not warn/fail when invoked via a path (like bin/pip) which can be checked using sys.argv[0]. The edge-case I see here is: on an OS where sys.argv[0] is something different from what is used to invoke the script. AFAICT, it's not a problem on Windows or MacOS 10.13 (High Sierra).

This should be a good thing to do, regardless of whether we decide to remove the pip* wrappers. Am I missing something here?

[pradyunsg]

Personally, I think a big source of confusion noted here is the mismatching pip vs python -m pip.

ISTM this is something that can be checked using sys.executable and PATH. Additionally, pip should not warn/fail when invoked via a path (like bin/pip) which can be checked using sys.argv[0]. The edge-case I see here is: on an OS where sys.argv[0] is something different from what is used to invoke the script. AFAICT, it's not a problem on Windows or MacOS 10.13 (High Sierra).

This should be a good thing to do, regardless of whether we decide to remove the pip* wrappers. Am I missing something here?

[pfmoore]

I think what you're missing here is that mostly, the pip executables in question are ones shipped by the distro, which pip can't update (for one reason or another - permissions, installed in a location other than where a normal pip install goes, PATH issues around user vs system installs, etc).

Personally, I suspect that this isn't a problem that we (pip) can solve, simply because the wrapper script that's messing things up isn't owned by us. That's why my inclination is to stop shipping the wrappers, and avoid adding to the confusion - make it clearly a case of "to run pip, use python -m pip, if you use a pip command, talk to whoever delivered it, it's not us". My second preference is to just ship an unversioned pip wrapper, and direct people hitting the issue to their distro vendor (we could write a "detect the config" script that we could ask people to run when they hit an issue like this - but by definition we can't ship it with pip because it's diagnosing issues where pip won't run correctly...) My third preference is the same, but we continue shipping versioned wrappers on Linux only (I believe they promote the bad practice of having multiple versions of Python on PATH, but I don't see how we're ever going to get Linux users to stop doing that).

But to answer your "regardless of whether we decide..." point, yes, writing a script that tries to diagnose the user's environment and report on misconfigurations and problems would be really useful. It's just that shipping it with pip would be of limited use, because often people wouldn't be able to run it in precisely the cases they need it. We could link to a standalone script from the docs, though.

[pfmoore]

I think what you're missing here is that mostly, the pip executables in question are ones shipped by the distro, which pip can't update (for one reason or another - permissions, installed in a location other than where a normal pip install goes, PATH issues around user vs system installs, etc).

Personally, I suspect that this isn't a problem that we (pip) can solve, simply because the wrapper script that's messing things up isn't owned by us. That's why my inclination is to stop shipping the wrappers, and avoid adding to the confusion - make it clearly a case of "to run pip, use python -m pip, if you use a pip command, talk to whoever delivered it, it's not us". My second preference is to just ship an unversioned pip wrapper, and direct people hitting the issue to their distro vendor (we could write a "detect the config" script that we could ask people to run when they hit an issue like this - but by definition we can't ship it with pip because it's diagnosing issues where pip won't run correctly...) My third preference is the same, but we continue shipping versioned wrappers on Linux only (I believe they promote the bad practice of having multiple versions of Python on PATH, but I don't see how we're ever going to get Linux users to stop doing that).

But to answer your "regardless of whether we decide..." point, yes, writing a script that tries to diagnose the user's environment and report on misconfigurations and problems would be really useful. It's just that shipping it with pip would be of limited use, because often people wouldn't be able to run it in precisely the cases they need it. We could link to a standalone script from the docs, though.

[ncoghlan]

See pypa/python-packaging-user-guide#396 for discussion of turning some recommended troubleshooting steps into an executable recipe.

[ncoghlan]

See pypa/python-packaging-user-guide#396 for discussion of turning some recommended troubleshooting steps into an executable recipe.

[dstufft]

That wrapper script is sort of owned by us. They're just using the one that we're generating, it'll just take a long time for a change to percolate out.

[dstufft]

That wrapper script is sort of owned by us. They're just using the one that we're generating, it'll just take a long time for a change to percolate out.

[pfmoore]

What I was getting at is that they don't install it where we expect them to (AFAIK, that's why sudo pip install -U pip breaks with the wrapper still looking for pip.main). There's also extra logic that we may need to put into the wrapper to handle disentangling systems that have pip installed system-wide as well as in user-site, and that is arguably a bug in our current scripts - although as far as I know the situation is muddled enough that I've never seen a good example of how to reproducibly get into that state...

[pfmoore]

What I was getting at is that they don't install it where we expect them to (AFAIK, that's why sudo pip install -U pip breaks with the wrapper still looking for pip.main). There's also extra logic that we may need to put into the wrapper to handle disentangling systems that have pip installed system-wide as well as in user-site, and that is arguably a bug in our current scripts - although as far as I know the situation is muddled enough that I've never seen a good example of how to reproducibly get into that state...

[pradyunsg]

@ncoghlan Thanks for that cross-reference to that issue. :)

@pfmoore That makes sense and makes me feel that separating the wrappers from pip the package would be useful; whether it's complete removal or moving it out to a pip-cli package.

[pradyunsg]

@ncoghlan Thanks for that cross-reference to that issue. :)

@pfmoore That makes sense and makes me feel that separating the wrappers from pip the package would be useful; whether it's complete removal or moving it out to a pip-cli package.

[pradyunsg]

I tried to come up with something that satisfies (nearly) all the concerns raised in this issue with the status-quo. I think a good position to end up in would be:

• pip does not provide any CLI wrappers; only supports python -m pip
• pip-cli mimics python -m pip in-process as a pip wrapper
• Include pip-cli in virtualenv's installed-by-default packages

Reasoning for choices here:

• Ease-of-use inside a virtualenv; it's still pip.
• Windows upgrade UX improves
  • pip install --upgrade pip will work on Windows 🎉
  • would still need to do python -m pip install --upgrade pip-cli though
    • slightly less of a problem since updates to that might be much less frequent
• Existing documentation says "pip install ..." which will still work in venvs.
• Not providing of pipX and pipX.Y
  • they're redundant in a virtualenv + issues listed in OP; they don't scale well
  • kinda sorta avoid conflicting with the distro's executables to an extent (when the distro doesn't provide unqualified variants)

Inputs and comments welcome on the above.

---

Other Notes:

• I don't see any compelling arguments for pipX and pipX.Y. They should probably get deprecated/removed.
• Related: #4625 (Overwriting existing wrappers)
  • relevant here: this would make pip-cli installation behave nicer with the distro-installed pips
• Having the "sanity check" checking equivalence of python -m pip and pip, shipped as a part of pip-cli sounds tempting to me.

[pradyunsg]

I tried to come up with something that satisfies (nearly) all the concerns raised in this issue with the status-quo. I think a good position to end up in would be:

• pip does not provide any CLI wrappers; only supports python -m pip
• pip-cli mimics python -m pip in-process as a pip wrapper
• Include pip-cli in virtualenv's installed-by-default packages

Reasoning for choices here:

• Ease-of-use inside a virtualenv; it's still pip.
• Windows upgrade UX improves
  • pip install --upgrade pip will work on Windows 🎉
  • would still need to do python -m pip install --upgrade pip-cli though
    • slightly less of a problem since updates to that might be much less frequent
• Existing documentation says "pip install ..." which will still work in venvs.
• Not providing of pipX and pipX.Y
  • they're redundant in a virtualenv + issues listed in OP; they don't scale well
  • kinda sorta avoid conflicting with the distro's executables to an extent (when the distro doesn't provide unqualified variants)

Inputs and comments welcome on the above.

---

Other Notes:

• I don't see any compelling arguments for pipX and pipX.Y. They should probably get deprecated/removed.
• Related: #4625 (Overwriting existing wrappers)
  • relevant here: this would make pip-cli installation behave nicer with the distro-installed pips
• Having the "sanity check" checking equivalence of python -m pip and pip, shipped as a part of pip-cli sounds tempting to me.

[pfmoore]

@pradyunsg Overall, this looks like a good plan to me.

[pfmoore]

@pradyunsg Overall, this looks like a good plan to me.

[ncoghlan]

+1 for @pradyunsg's plan from me.

The one case of pipX I'm aware of that does see use is pip3 install --user ... on Linux systems, but changing that to python3 -m pip install --user ... instead isn't too much of a burden.

If distros decided they wanted to add the wrapper script back to their python2-pip and python3-pip packages for backwards compatibility reasons, I also think we'd be OK with that.

[ncoghlan]

+1 for @pradyunsg's plan from me.

The one case of pipX I'm aware of that does see use is pip3 install --user ... on Linux systems, but changing that to python3 -m pip install --user ... instead isn't too much of a burden.

If distros decided they wanted to add the wrapper script back to their python2-pip and python3-pip packages for backwards compatibility reasons, I also think we'd be OK with that.

[pradyunsg]

I think the strategy would be pretty straightforward for pipX, pipX.Y:

1. Release n
   • Deprecate pipX, pipX.Y; shows a warning about removal when they are used.
2. Release n+1 (or n+2)?
   • Remove pipX, pipX.Y from pip.

It's straightforward enough that I think we can let n here be 18.0; scheduled sometime next month. The only thing here would be: how long do we run the deprecation? I'm on the fence on that.

---

For pip, it gets more interesting. I'm pretty sure we don't want this change to be overly disruptive for user workflows and want provide a smooth way to transition to pip-cli. One thing I think we should do here is add a special case so that upgrading pip's pip doesn't overwrite pip-cli's pip during the transition but vice-versa works.

Ideally, we'd have some sort of "beta" period for using pip-cli where we could ask users to test out using pip-cli. That'd help iron out issues before we deprecate pip in pip.

1. Release n
   • Deprecate pip of pip.
   • Add pip-cli to virtualenv; ensuring it gets installed after pip.
   • Suggest users to switch to pip-cli.
2. Release n+2
   • Remove pip from pip.

I think the 2 release cycles should be enough time to gather user feedback on this change and react appropriately. I do think the release before the one we remove pip in should say "the pip wrapper will be dropped in the next release" instead of "in a future release" like they usually do.

---

Now, I'm not sure if and where we should include some variant of the sanity-check/debugging information?

• in pip's pip? In Release n + 1?
• in pip-cli's pip; in the run up to Release n?

[pradyunsg]

I think the strategy would be pretty straightforward for pipX, pipX.Y:

1. Release n
   • Deprecate pipX, pipX.Y; shows a warning about removal when they are used.
2. Release n+1 (or n+2)?
   • Remove pipX, pipX.Y from pip.

It's straightforward enough that I think we can let n here be 18.0; scheduled sometime next month. The only thing here would be: how long do we run the deprecation? I'm on the fence on that.

---

For pip, it gets more interesting. I'm pretty sure we don't want this change to be overly disruptive for user workflows and want provide a smooth way to transition to pip-cli. One thing I think we should do here is add a special case so that upgrading pip's pip doesn't overwrite pip-cli's pip during the transition but vice-versa works.

Ideally, we'd have some sort of "beta" period for using pip-cli where we could ask users to test out using pip-cli. That'd help iron out issues before we deprecate pip in pip.

1. Release n
   • Deprecate pip of pip.
   • Add pip-cli to virtualenv; ensuring it gets installed after pip.
   • Suggest users to switch to pip-cli.
2. Release n+2
   • Remove pip from pip.

I think the 2 release cycles should be enough time to gather user feedback on this change and react appropriately. I do think the release before the one we remove pip in should say "the pip wrapper will be dropped in the next release" instead of "in a future release" like they usually do.

---

Now, I'm not sure if and where we should include some variant of the sanity-check/debugging information?

• in pip's pip? In Release n + 1?
• in pip-cli's pip; in the run up to Release n?

[pradyunsg]

@pypa/pip-committers Let's deprecate pipX and pipX.Y in pip 18.1?

[pradyunsg]

@pypa/pip-committers Let's deprecate pipX and pipX.Y in pip 18.1?

[dstufft]

Is our plan just to tell people to use python -m pip then?

[dstufft]

Is our plan just to tell people to use python -m pip then?

[pradyunsg]

Yes. That's what we'd have.

[pradyunsg]

Yes. That's what we'd have.

[ncoghlan]

It's a little bit trickier than that, since "python" may not refer to the right thing. That means any deprecation warning will need some heuristics based on sys.executable, shutil.which and the running platform to decide what a suitable replacement would be.

[ncoghlan]

It's a little bit trickier than that, since "python" may not refer to the right thing. That means any deprecation warning will need some heuristics based on sys.executable, shutil.which and the running platform to decide what a suitable replacement would be.