Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define all old pip entrypoints #7498

Closed
chrahunt opened this issue Dec 19, 2019 · 14 comments · Fixed by #7499
Closed

Define all old pip entrypoints #7498

chrahunt opened this issue Dec 19, 2019 · 14 comments · Fixed by #7499

Comments

@chrahunt
Copy link
Member

@chrahunt chrahunt commented Dec 19, 2019

What's the problem this feature will solve?

Tons of users are experiencing problems related to #5599, as evidenced by the traffic that we're getting to related issues (past 2 weeks):

Link Views Unique Visitors
#5447 11,368 9,005
#5221 3,252 2,733
#5240 2,398 2,068
#5599 2,247 1,857

This probably leads to a lot of negative feelings not just for pip, but the Python packaging ecosystem in general.

Describe the solution you'd like

Move the current entrypoint from pip._internal.main:main to pip._internal.cli.main:main (or some other location).

Create a function for every known past entrypoint:

  • pip._internal.main:main
  • pip._internal:main
  • pip:main

At the very least the functions should invoke the correct current pip entrypoint. This alone would prevent a lot of needless suffering and should be easy to implement and test.

Later we could also consider printing a warning, but I'd rather handle that as a separate issue since this is useful on its own.

Alternative Solutions

  • do nothing

Additional context

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Dec 19, 2019

I agree with this as a stopgap measure, but I'm a strong -1 on retaining such "legacy" entry points indefinitely (unless we change our official position on "no supported API").

Personally, I would rather include a warning right now, saying something along the lines of "you are calling the wrong internal entry point for pip (possibly as a result of using an old wrapper script)". We can tidy up the message, and/or find a better long-term solution, later, but I feel that we should have something from the start.

The long-term solution will be hard to identify, as most of the time the failing wrappers appear to be distro-supplied, or the result of users with limited command line knowledge misunderstanding concepts like PATH. Both of which are reasonable issues, but out of our control. Maybe we should finally switch to making python -m pip the only supported way of invoking pip, and stop shipping the wrappers? It's hard to argue that the wrapper is for "convenience" if it's causing this level of problems for so many users. We could then have the warning suggest that users need to invoke pip that way.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Dec 19, 2019

Alternative solution: #3164.

This could be handled as a phased approach:

  1. Document that python -m pip is the supported way to run pip.
  2. Add a warning to the wrappers, saying that people should switch to python -m pip.
  3. Make that warning a hard error.
  4. Remove the wrappers altogether.

It would obviously need a long deprecation period, but I'm fine with that.

In the short term, any issues with wrappers failing can be handled by simply advising the user to switch to python -m pip. That's a policy that could be adopted now.

@uranusjr

This comment has been minimized.

Copy link
Member

@uranusjr uranusjr commented Dec 19, 2019

I’m a little skeptical wrt deprecating the pip entrypoint. The main source of the problem here is entrypoints from Linux distributors. I kind of wonder whether they would honour pip removing entrypoints, or go on to generate their own anyway. We wouldnt solve much of the problem if they do (and maybe creating more since there would then be two different entrypoint flavours in the market).

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Dec 19, 2019

The main source of the problem here is entrypoints from Linux distributors.

Note: You mean "wrapper" rather than "entrypoint" here, but your point is valid.

I've been very careful as a Windows user to avoid mentioning distros, but as you raised it, I think it's relevant. Many of the issues we see with the main() API appear to come from how distributions ship wrappers as part of their system repackaging of pip. Without some sort of assistance or support from the distros, I don't think there's any way we can cleanly solve this issue (short of taking the view that the distros are an irresistable force and we simply have to make the main() API supported at least to the level that they require, regardless of our wishes).

To be fair, though, pip's policy has always been to leave the system pip to the distro (although user installs are a sticking point here). But many users don't follow that advice as well as we'd like - not least because doing so (as far as I can tell) needs a reasonable understanding of how PATH works, which not all of our users have.

If we deprecated the pip wrapper, we could at least say to anyone using a wrapper "either reproduce the issue using python -m pip or discuss with your distro support channel". Hardly ideal, but at least a clear separation of responsibilities...

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Dec 20, 2019

I agree with making this a stop-gap solution (I'd assumed that was the case, since that's basically how I was personally viewing this).

It would obviously need a long deprecation period, but I'm fine with that.

IMO, we should consider reaching out to distros, to get them to fix their wrappers -- in their current distro releases as bug fixes, or in the next "major" version. That would help inform the length of this deprecation period.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Dec 20, 2019

IMO, we should consider reaching out to distros

+1. Does anyone have a good means of doing this (and of tracking it, we should monitor responses and progress on this, and link from here)?

@chrahunt

This comment has been minimized.

Copy link
Member Author

@chrahunt chrahunt commented Dec 20, 2019

Is this actually a problem with distro wrappers? I assumed that the distro wrappers were consistent with the system-managed pip and the issue in most cases was either:

  • A user-scheme install, but with
    • ~/.local/bin later on PATH
    • ~/.local/bin not on PATH
    • ~/.local/bin earlier on PATH, but the shell has hashed pip
  • A global install into the same site-packages as the system-managed pip, but the "scripts" dir used is after the one used by the system-managed pip in PATH
  • A global install into a different site-packages and "scripts" dir as the system-managed pip, but which has sys.path precedence to the system-managed pip, but the system-managed pup's script dir is ahead on PATH
  • Equivalent on Windows

None of which I think are distro issues.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Dec 20, 2019

None of which I think are distro issues.

Maybe you're right. But the whole point of having this issue at all is that we don't want to categorise these problems as user issues (after all, we're putting a lot of effort into working round the fact that some users don't understand how PATH works). And yet this clearly isn't a pip issue, as the wrapper pip installs works with the version of pip that pip installs - it's not our fault if the user executes a different wrapper. So there's not really anyone left apart from the distros 🙁

The point of involving the distros is (as far as I can see) that they are better able to help users with general "command line UX" issues. Maybe they won't want to, and maybe they are stricter in their view that knowing how to use the command line is the user's problem. But at least we can ask.

Maybe this is something we should be referring to the funded work on pip's UX that I believe is under consideration. I've no idea where that is up to right now, but I'd assume that the UX specialists who would be involved in that would have good insight into this sort of thing.

@oldium

This comment has been minimized.

Copy link

@oldium oldium commented Jan 4, 2020

Big thanks to @chrahunt that I will not need to face this issue again and again on multiple systems and installations. 👍🎉

@oldium

This comment has been minimized.

Copy link

@oldium oldium commented Jan 4, 2020

For others wanting to try this locally before official release – this does the trick python -m pip install --user --upgrade git+https://github.com/pypa/pip.git.

The warning is very confusing to me, because I actually just updated the pip package. I did nothing wrong, still I see the warning. Pip is basically warning that it uses pip... 😕

Anyway, good move towards having this definitely fixed. 👍

@chrahunt

This comment has been minimized.

Copy link
Member Author

@chrahunt chrahunt commented Jan 5, 2020

Hi @oldium, can you explain what you mean about the warning being confusing? What is your current understanding of the situation given what was output by pip? We can always improve it.

@oldium

This comment has been minimized.

Copy link

@oldium oldium commented Jan 5, 2020

Hi, I will illustrate it on an example:

#> pip --version
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
pip 20.0.dev0 from C:\Users\oldium\AppData\Roaming\Python\Python38\site-packages\pip (python 3.8)

I have latest python 3.8.1, pip during installation of some package wrote to me that there is a newer version. Ok, I updated pip (I updated to latest master), but now pip complains with much bigger warning that there is something wrong. Also the requested version output is unreadable because of this.

I find it really useful to have pip shortcut, because it is harder to remember the sequence python -m pip, I had to search Google almost every time, because I use pip very rarely. I see the warning confusing, because it only informs me that pip is calling some internal API. I can imagine that public API during development prints warnings, but I see no reason to bother user about something that he cannot influence - in the release version and internal API usage. This does not make sense to me.

@pradyunsg

This comment has been minimized.

Copy link
Member

@pradyunsg pradyunsg commented Jan 5, 2020

@oldium What's the output of the following commands for you?

> where pip
> type "<copy-paste the output from the last command here>"
@oldium

This comment has been minimized.

Copy link

@oldium oldium commented Jan 5, 2020

#> where.exe pip
C:\Python38\Scripts\pip.exe
C:\Users\oldium\AppData\Roaming\Python\Python38\Scripts\pip.exe

Note: I updated pip with the --user flag, because I did not want to modify the installed version (and therefore use console with administrator rights).

pip gets executed from C:\Python38\Scripts, because it is found in the system's PATH environment variable, which is prepared by Python installer (I used Chocolatey to install it). User's C:\Users\oldium\AppData\Roaming\Python\Python38\Scripts is not on PATH by default, I had to add it manually. I logically did this on the user's PATH environment variable, but that one is taken after the system's one, so even when I have the updated pip.exe binary in AppData folder, it does not get executed. So I have a classic mismatch of old pip executable and new pip python site-package.

Annoying is that when I use admin rights (not everybody has them) and add my user's AppData Python38 folder to the beginning of the system PATH environment variable, this breaks with next release of Python39, because I would have to manually update the system PATH again (and remember that I have to do it). And I expect that once I will not be ready, the pip API breakage strikes again (as it did several times in past).

That was Windows. Ubuntu 18.04 adds ~/.local/bin to the beginning of the PATH, if the bin folder exists, so it might take the updated pip binary. The .local/bin directory does not exist initially, so if you simply update pip and try to use it immediately from the same shell, you will face the same issue actually...

During past several years I have not found a good solution to this problem, neither on Windows and neither on Linux. The simplest would be if pip had a stable internal API entry point. And I prefer to use pip shorthand, because remembering python -m pip is harder (I remember it now, but that is a result of multiple frustrations in past spending searching Google for the reason of pip breakage and the right syntax to use the module directly).

Why is pip executable simply not trying to call python -m pip anyway?

hartb added a commit to hartb/apex that referenced this issue Jan 8, 2020
The 'main' function was moved between pip 19.2.3 and 19.3, which
causes '--pyprof' build to fail with (at least) pip 19.3 and 19.3.1:

    Traceback (most recent call last):
      File "setup.py", line 37, in <module>
        pipmain(["install"] + required_packages)
    TypeError: 'module' object is not callable

A permanent solution for this might require rewriting the '--pyprof'
install section to use 'python -m pip', but this change just adjusts
the import to tolerate the 19.3 change.

See:

- pypa/pip@09fd200
- pypa/pip#7498
- pypa/pip#7499
@lock lock bot added the S: auto-locked label Feb 4, 2020
@lock lock bot locked as resolved and limited conversation to collaborators Feb 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

5 participants
You can’t perform that action at this time.