Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError in system pip wrappers after an upgrade #5599

Open
pfmoore opened this issue Jul 13, 2018 · 117 comments
Open

ImportError in system pip wrappers after an upgrade #5599

pfmoore opened this issue Jul 13, 2018 · 117 comments
Labels

Comments

@pfmoore
Copy link
Member

@pfmoore pfmoore commented Jul 13, 2018

Overview

After upgrading to pip 10 or higher, many users are encountering error like

These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we'll try to help you solve your issue, this is not the "fault" of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

  1. Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
  2. Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
  3. Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

Debugging the Issue

Before trying to work out what's going on, it's critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That's often in the traceback you get, but if it isn't, you can use OS tools like which, or Python's shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can't, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you're usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that's why it imports pip.main) under a Python interpreter that sees a copy of pip that's version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that's where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution's pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it's quite likely with problems of this nature that you'll need to modify system-supplied files or settings. You do so at your own risk. If you're not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There's only so much that can be achieved remotely.

Also, the pip developers don't provide any guarantees that advice in the comments on this issue is correct, or that it won't damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue:
#5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

@denisglotov
Copy link

@denisglotov denisglotov commented Aug 5, 2018

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side.
To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

@randomdudethathelpsyou
Copy link

@randomdudethathelpsyou randomdudethathelpsyou commented Aug 13, 2018

I am pretty new to coding, however I managed to fix it in a kind of odd way.
My problem was that there was nothing in my /usr/bin/ folder regarding pip.
to fix this I had my buddy upload the get-pip.py program to raspberry pi, from which I downloaded the code to my computer and ran it. It then over-rode the old pip program and installed the new one.

Here's the code we used:
working computer to pi: scp /whatever_your_directory_is/get-pip.py pi@pi's_address:/pi's_directory
pi to my computer (the last period is important): scp pi@10.200.130.65:/pi's_directoryi/get-pip.py .
to download pip: sudo python get-pip.py

Sorry if it's weird, but that's what got it working for me

@veneman
Copy link

@veneman veneman commented Aug 17, 2018

After upgrading pip I ran into the error ImportError: cannot import name 'py31compat' when trying to import pyarrow. I have a conda installation and after rolling back pip to version 9 on all of my environments I still ran into the error.

The fix for me was to change line 72 in site-packages/pkg_resources/__init__.py from from . import py31compat to from tempfile import TemporaryDirectory.

setuptools/py31compat.py is a workaround for older versions of Python that don't have TemporaryDirectory. Since I'm running Python 3.6 this change eliminates the need to use py31compat.

Things seem to be working after this change but I'm not that confident it'll keep working down the road.

@ErichBSchulz
Copy link

@ErichBSchulz ErichBSchulz commented Aug 17, 2018

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit :-/

@pfmoore
Copy link
Member Author

@pfmoore pfmoore commented Aug 17, 2018

@ErichBSchulz We're aware of that - see #5346.

Maybe I should have more explicitly said "Only ever use your system package manager to upgrade your system pip". The more detailed comment in that bullet point makes it clearer, but I've edited the post to make it explicit.

@MannyC
Copy link

@MannyC MannyC commented Aug 19, 2018

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

@PieterGit
Copy link

@PieterGit PieterGit commented Aug 27, 2018

I would add this advice of not upgrading your system pip in the documentation.
I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page:
https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:

python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip 

After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip
The request to update pip in debian repo can be found here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

@borismo
Copy link

@borismo borismo commented Sep 10, 2018

pip3 install --user pip seems to break my system pip.

Isn't it a good idea to install the latest version of pip in the user's home directory?

@abhiTronix
Copy link

@abhiTronix abhiTronix commented Sep 12, 2018

The Ubuntu's python-pip-9.0.1 installed(via apt get python-pip) is conflicting with Latest pip-18.0 provided by pypa. So sudo rm -rf ~/.local/lib/python2.7/site-packages/(Beware!) removes the ubuntu's python-pip and resolves the problem. Only one pip can exist on the system. Thanks @denisglotov

Also Upgrading pip doesn't help as it does'nt overwrite the preexisting installed python-pip but instead create another copy of pip in different directory causing the conflict.

@Ark-kun
Copy link

@Ark-kun Ark-kun commented Sep 13, 2018

My problem was installing pip in ~/.local/ and having at the end of $PATH.

@ddombrowsky
Copy link

@ddombrowsky ddombrowsky commented Sep 21, 2018

I wonder how npm and cpan solve this problem? In any case, I just uninstalled system pip3 and created my own:

#!/bin/sh

exec python3 -m pip "$@"

I'm the only user running it on this system anyway. Victory!

@lemon-doge
Copy link

@lemon-doge lemon-doge commented Nov 5, 2020

Seems like thats my case:

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

But what do i do to fix the problem precisely? Do i need to reset the PATH? Current echo PATH output is /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/Library/Frameworks/Mono.framework/Versions/Current/Commands

Mac OS Catalina 10.15.6
what do I do? pls help

@Newtoxton
Copy link

@Newtoxton Newtoxton commented Nov 7, 2020

sudo apt-get --reinstall install python3-setuptools python3-wheel python3-pip
sudo -H pip3 install --upgrade pip

Worked for me

This worked for me after multiple tries on other methods.

@lemon-doge
Copy link

@lemon-doge lemon-doge commented Nov 8, 2020

@KTibow
Copy link

@KTibow KTibow commented Nov 8, 2020

@lemon-doge linux has it. Install homebrew for macs as an alternative.

@Newtoxton
Copy link

@Newtoxton Newtoxton commented Nov 8, 2020

Thanks for your response) Are you using mac or linux? Cause I get something like this sudo: apt-get: command not found. Im on mac os Catalina 10.15.6

________________________________ От: Newtoxton notifications@github.com Отправлено: 8 ноября 2020 г. 1:22 Кому: pypa/pip pip@noreply.github.com Копия: Татьянченко Дмитрий Денисович ddtatyanchenko@edu.hse.ru; Comment comment@noreply.github.com Тема: Re: [pypa/pip] ImportError in system pip wrappers after an upgrade (#5599) sudo apt-get --reinstall install python3-setuptools python3-wheel python3-pip sudo -H pip3 install --upgrade pip Worked for me This worked for me after multiple tries on other methods. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#5599 (comment)>, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQ4INI6GIIU47LX367TZLA3SOXCDLANCNFSM4FJ4EQWA.

Using Linux on a virtual machine

@andy5995
Copy link

@andy5995 andy5995 commented Nov 25, 2020

After upgrading using the --user option (as an unprivileged user without sudo), I got this warning:

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.

The odd thing is that:

andy@oceanus:~/.local/bin$ which pip
/home/andy/.local/bin/pip
andy@oceanus:~/.local/bin$ which pip3
/home/andy/.local/bin/pip3
andy@oceanus:~/.local/bin$

After I switched to that path and used ./pip3..., the warning went away. It turned out that pip3 was being run from /usr/bin, even though ~/.local/bin has precedence in my PATH var.

To fix the problem, as root, I made sure pip was installed with the --user option, upgraded it, then ran apt-get remove python-pip python3-pip.

And I edited /root/.bashrc to add the line:

PATH=$HOME/.local/bin:$PATH

My OS is Debian Buster.

@pradyunsg pradyunsg unpinned this issue Dec 15, 2020
imphil added a commit to imphil/opentitan that referenced this issue Dec 15, 2020
Avoid using an old "pip3" wrapper script, as indicated by this warning
we're getting in CI:

```
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see pypa/pip#5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
```

Signed-off-by: Philipp Wagner <phw@lowrisc.org>
imphil added a commit to lowRISC/opentitan that referenced this issue Dec 15, 2020
Avoid using an old "pip3" wrapper script, as indicated by this warning
we're getting in CI:

```
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see pypa/pip#5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
```

Signed-off-by: Philipp Wagner <phw@lowrisc.org>
@aktaseren
Copy link

@aktaseren aktaseren commented Dec 18, 2020

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side.
To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

I am having the same pip issue for a PyTorch implementation. Since I am unfortunately using macOS latest version BigSur, I can not apply the solutions suggested here.

Can you please guide me on how I can solve this issue on macOS?

My pip version and the location is below:

% python -m pip --version
pip 20.1.1 from /Users/455832/opt/anaconda3/lib/python3.7/site-packages/pip (python 3.7)

My issue is below after running the lines below:

solver = main(hyperparameters(defaults))
accuracy, precision, recall, f_score = solver.test()
-----------------------
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-29-a896b364722b> in <module>
----> 1 solver = main(hyperparameters(defaults))
      2 accuracy, precision, recall, f_score = solver.test()

~/opt/anaconda3/lib/python3.7/site-packages/pip/__init__.py in main(args)
     16     from pip._internal.utils.entrypoints import _wrapper
     17 
---> 18     return _wrapper(args)

~/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/utils/entrypoints.py in _wrapper(args)
     29         "running pip directly.\n"
     30     )
---> 31     return main(args)

~/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/main.py in main(args)
     58 
     59     try:
---> 60         cmd_name, cmd_args = parse_command(args)
     61     except PipError as exc:
     62         sys.stderr.write("ERROR: {}".format(exc))

~/opt/anaconda3/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py in parse_command(args)
     65     #  general_options: ['--timeout==5']
     66     #  args_else: ['install', '--user', 'INITools']
---> 67     general_options, args_else = parser.parse_args(args)
     68 
     69     # --version

~/opt/anaconda3/lib/python3.7/optparse.py in parse_args(self, args, values)
   1367         over after parsing options.
   1368         """
-> 1369         rargs = self._get_args(args)
   1370         if values is None:
   1371             values = self.get_default_values()

~/opt/anaconda3/lib/python3.7/optparse.py in _get_args(self, args)
   1351             return sys.argv[1:]
   1352         else:
-> 1353             return args[:]              # don't modify caller's list
   1354 
   1355     def parse_args(self, args=None, values=None):

TypeError: 'hyperparameters' object is not subscriptable

@KTibow
Copy link

@KTibow KTibow commented Dec 18, 2020

@aktaseren Please use the code button. image

@aktaseren
Copy link

@aktaseren aktaseren commented Dec 20, 2020

@KTibow Sorry I had forgotten it. I editted for the code part. Please share your suggestion/s if you have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.