New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportError in system pip wrappers after an upgrade #5599

Open
pfmoore opened this Issue Jul 13, 2018 · 15 comments

Comments

Projects
None yet
@pfmoore
Member

pfmoore commented Jul 13, 2018

Overview

After upgrading to pip 10 or higher, many users are encountering variations of the error ImportError: cannot import name 'main'. These are caused by an incorrect attempt to upgrade pip, which has typically resulted in (parts of) multiple versions of pip being installed in the same Python installation, and those parts being incompatible.

It should be noted that these issues are invariably not problems with pip itself, but are due to incorrect use of pip, or unexpected interactions with system scripts that are not controlled by pip. So while we'll try to help you solve your issue, this is not the "fault" of pip, and you will have to be prepared to do at least some of the debugging and fixes on your own.

General Advice

First, some general advice. It is assumed that anyone encountering issues will have failed to follow some part of this advice. Listing these items here is not intended to imply that "it's your fault and we won't help", but rather to help users to identify what went wrong, so that they can work out what to do next more easily.

  1. Only ever use your system package manager to upgrade the system pip. The system installed pip is owned by the distribution, and if you don't use distribution-supplied tools to manage it, you will hit problems. Yes, we know pip says "you should upgrade with pip install -U pip" - that's true in a pip-managed installation, ideally distributions should patch this message to give appropriate instructions in the system pip, but they don't. We're working with them on this, but it's not going to happen soon (remember, we're looking at cases where people are upgrading old versions of pip here, so patches to new versions won't help).
  2. Never use sudo with pip. This follows on from the first point. If you think you need to use sudo, you're probably trying to modify a distribution-owned file. See point 1.
  3. Prefer to use --user. By doing this, you only ever install packages in your personal directories, and so you avoid interfering with the system copy of pip. But there are PATH issues you need to be aware of here. We'll cover these later. Put simply, it's possible to follow this advice, and still hit problems, because you're not actually running the wrapper you installed as --user.

Debugging the Issue

Before trying to work out what's going on, it's critically important that you understand precisely what scripts you are running and what versions of pip the relevant Python interpreter is using.

First, identify the full absolute path of the executable script that you are running. That's often in the traceback you get, but if it isn't, you can use OS tools like which, or Python's shutil.which to identify the right script. Watch out for your shell confusing the issue, with aliases or potentially stale hashed commands.

Once you have identified the script, make sure you can reproduce the problem using the absolute path to that script. If you can't, you probably found the wrong script, so check again.

Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).

Now run python -m pip --version. This will give you the exact version and location of the installation of pip that your Python is seeing.

At this point, you're usually done - the fundamental cause of all these problems is running a wrapper script which is written expecting to see a version of pip older than pip 10 (that's why it imports pip.main) under a Python interpreter that sees a copy of pip that's version 10 or later.

Fixing the Issue

The problem, of course, is fixing the issue. And that's where you really are on your own. If you have changed your system installation, you really need to put it back the way that the distribution installed it. That may well require an uninstall and reinstall of your distribution's pip package. Reinstalling is easy, of course - but uninstalling may require manually removing incorrect files. Or you may be able to force-reinstall with your distribution package manager, simply overwriting the incorrect files. Of course, this reverts you to the system-supplied version of pip. If you need a newer version, you should ask your distribution vendor, or use something like virtualenv to install it independently of your system packages.

It may be that you're simply running the "wrong" wrapper script. Maybe you did a --user install of a new version of pip, but your PATH is set to run the system version of the wrapper rather than the user-local one installed with pip. In that case, you can simply fix your PATH. That's usually the issue for people who do pip install --user --upgrade pip and get the pip.main error.

As already noted, python -m pip is a reliable workaround, at the cost of using a more verbose command to invoke pip.

Community Advice

The comments section of this issue is open for people to discuss specific problems, and to share potential solutions. Just to be clear, it's quite likely with problems of this nature that you'll need to modify system-supplied files or settings. You do so at your own risk. If you're not comfortable modifying your OS, or running as root, you should seek expert advice. To put it another way, if by following suggestions given here, you break your system, you get to keep the pieces. There's only so much that can be achieved remotely.

Also, the pip developers don't provide any guarantees that advice in the comments on this issue is correct, or that it won't damage your system. Again, use at your own risk.

Related Issues

The following issues have been reported which are related to this issue:
#5240, #5221, #5588, #5495, #5493, #5487, #5447, #5432, #5373, #5326, #5318, #5253

@denisglotov

This comment has been minimized.

denisglotov commented Aug 5, 2018

The bug happens when not using virtualenv, using virtualenv fixes the problem on my side.
To fix the duplicate pip problem (described above) and downgrade to original pip version, I just rm -rf ~/.local/lib/python2.7/site-packages/ (beware!).

@randomdudethathelpsyou

This comment has been minimized.

randomdudethathelpsyou commented Aug 13, 2018

I am pretty new to coding, however I managed to fix it in a kind of odd way.
My problem was that there was nothing in my /usr/bin/ folder regarding pip.
to fix this I had my buddy upload the get-pip.py program to raspberry pi, from which I downloaded the code to my computer and ran it. It then over-rode the old pip program and installed the new one.

Here's the code we used:
working computer to pi: scp /whatever_your_directory_is/get-pip.py pi@pi's_address:/pi's_directory
pi to my computer (the last period is important): scp pi@10.200.130.65:/pi's_directoryi/get-pip.py .
to download pip: sudo python get-pip.py

Sorry if it's weird, but that's what got it working for me

@veneman

This comment has been minimized.

veneman commented Aug 17, 2018

After upgrading pip I ran into the error ImportError: cannot import name 'py31compat' when trying to import pyarrow. I have a conda installation and after rolling back pip to version 9 on all of my environments I still ran into the error.

The fix for me was to change line 72 in site-packages/pkg_resources/__init__.py from from . import py31compat to from tempfile import TemporaryDirectory.

setuptools/py31compat.py is a workaround for older versions of Python that don't have TemporaryDirectory. Since I'm running Python 3.6 this change eliminates the need to use py31compat.

Things seem to be working after this change but I'm not that confident it'll keep working down the road.

@ErichBSchulz

This comment has been minimized.

ErichBSchulz commented Aug 17, 2018

The advice above:

  Only ever use your system package manager to upgrade pip.

conflicts with the cheerful siren call:

You are using pip version 8.1.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

This seems to be some kind of tarpit :-/

@pfmoore

This comment has been minimized.

Member

pfmoore commented Aug 17, 2018

@ErichBSchulz We're aware of that - see #5346.

Maybe I should have more explicitly said "Only ever use your system package manager to upgrade your system pip". The more detailed comment in that bullet point makes it clearer, but I've edited the post to make it explicit.

@MannyC

This comment has been minimized.

MannyC commented Aug 19, 2018

I followed some instructions elsewhere and ran

sudo apt install python-pip
pip install --upgrade pip

and hit upon this issue.

python -m pip uninstall pip

fixed it, returning me to the system pip

I have to say, although I don't fully understand the complications involved, it doesn't feel like pip should self-immolate like this.

@PieterGit

This comment has been minimized.

PieterGit commented Aug 27, 2018

I would add this advice of not upgrading your system pip in the documentation.
I just followed these instructions and killed my raspbian pip and pip3 installs.

On the page:
https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date it lists python -m pip install --upgrade pip setuptools wheel , which is in conflict with recommentation 1 and 3 in this issue.

I think I restored that with:

python3 -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install  python3-setuptools python3-wheel python3-pip

python -m pip uninstall pip setuptools wheel
sudo apt-get --reinstall install python-setuptools python-wheel python-pip 

After that my raspbian PI zero pip and pip3 report the ancient 9.0.1 versions, which is the latests available according to https://tracker.debian.org/pkg/python-pip
The request to update pip in debian repo can be found here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901393

@borismo

This comment has been minimized.

borismo commented Sep 10, 2018

pip3 install --user pip seems to break my system pip.

Isn't it a good idea to install the latest version of pip in the user's home directory?

@abhiTronix

This comment has been minimized.

abhiTronix commented Sep 12, 2018

The Ubuntu's python-pip-9.0.1 installed(via apt get python-pip) is conflicting with Latest pip-18.0 provided by pypa. So sudo rm -rf ~/.local/lib/python2.7/site-packages/(Beware!) removes the ubuntu's python-pip and resolves the problem. Only one pip can exist on the system. Thanks @denisglotov

Also Upgrading pip doesn't help as it does'nt overwrite the preexisting installed python-pip but instead create another copy of pip in different directory causing the conflict.

@Ark-kun

This comment has been minimized.

Ark-kun commented Sep 13, 2018

My problem was installing pip in ~/.local/ and having at the end of $PATH.

@CameronAckermanSEL

This comment has been minimized.

CameronAckermanSEL commented Sep 14, 2018

I really don't want to sit here for hours wrestling with this. I upgraded from 8 using the recommended upgrade instructions from the command line and got jetted to 18.0 where my system is broken.

Can someone give me the idiots cliffnotes guide to unborking my system?

@ddombrowsky

This comment has been minimized.

ddombrowsky commented Sep 21, 2018

I wonder how npm and cpan solve this problem? In any case, I just uninstalled system pip3 and created my own:

#!/bin/sh

exec python3 -m pip "$@"

I'm the only user running it on this system anyway. Victory!

@axodd

This comment has been minimized.

axodd commented Sep 29, 2018

Related discussion (from April) with a variety of solutions from Stackoverflow community:
https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main
https://stackoverflow.com/questions/49964093
Let's hope #5346 will help minimize future impacts, as OS distributions alone may be too slow at preventing this issue from happening in their LTS and later releases..

@hawran

This comment has been minimized.

hawran commented Oct 18, 2018

...

Debugging the Issue

...
Second, work out which version of Python the script is using to run pip. You'll often be able to get that from the shebang line of the script. This can often be the trickiest problem, as wrapper scripts can take many forms depending on what tool created them. In the worst case, you can simply make an intelligent guess at this point.

Once you know which Python is running pip, you can run python -m pip to invoke pip. In most cases, this will work fine, as it's the wrapper causing the issue, and not pip itself. Running pip via python -m in this way is often a perfectly acceptable workaround for the issue (at least in the short term).
...

Well:

$ python -m pip
/usr/bin/python: No module named pip
$ python3 -m pip3
/usr/bin/python3: No module named pip3

OK, so let's try something else:

$ python3 -m pip uninstall pip3
Skipping pip3 as it is not installed.
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Wait,
'You are using pip version 10.0.1, however version 18.1 is available.'

What?

$ dpkg -l '*pip*'
...                                                                  Version                                   Architecture                              Description
+++-=========================================================================-=========================================-=========================================-=======================================================================================================================================================
ii  libpipeline1:amd64                                                        1.4.1-2                                   amd64                                     pipeline manipulation library
un  pppdcapiplugin                                                            <none>                                    <none>                                    (no description available)
$

So what am I supposed to do now?

PS I really consider the sentence:
'You are using pip version 10.0.1, however version 18.1 is available.' really, really confusing...

@hawran

This comment has been minimized.

hawran commented Oct 18, 2018

...

Update:
Finally I've managed to fix my system using the #5599 (comment).
Thank you very much!


Xubuntu

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:        16.04
Codename:       xenial
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment