pip 9.0.2 KeyError: 'pip._vendor.urllib3.contrib' #5079

vhaldemar · 2018-03-17T14:40:01Z

Pip version: 9.0.2
Python version: 2.7.3, 2.7.13
Operating system: Ubuntu 12.04, Ubuntu 16.04

In [1]: import urllib3.contrib

In [2]: import pip.pep425tags
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-2-9ae721442c45> in <module>()
----> 1 import pip.pep425tags

/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/__init__.py in <module>()
     43 from pip.utils import get_installed_distributions, get_prog
     44 from pip.utils import deprecation, dist_is_editable
---> 45 from pip.vcs import git, mercurial, subversion, bazaar  # noqa
     46 from pip.baseparser import ConfigOptionParser, UpdatingDefaultsHelpFormatter
     47 from pip.commands import get_summaries, get_similar_commands

/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/vcs/mercurial.py in <module>()
      7 from pip.utils import display_path, rmtree
      8 from pip.vcs import vcs, VersionControl
----> 9 from pip.download import path_to_url
     10 from pip._vendor.six.moves import configparser
     11 

/home/l<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/download.py in <module>()
     38 from pip.locations import write_delete_marker_file
     39 from pip.vcs import vcs
---> 40 from pip._vendor import requests, six
     41 from pip._vendor.requests.adapters import BaseAdapter, HTTPAdapter
     42 from pip._vendor.requests.auth import AuthBase, HTTPBasicAuth

/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py in <module>()
     96 
     97 from . import utils
---> 98 from . import packages
     99 from .models import Request, Response, PreparedRequest
    100 from .api import request, get, head, post, patch, put, delete, options

/home/<...>/.virtualenvs/other-sabt/local/lib/python2.7/site-packages/pip/_vendor/requests/packages.py in <module>()
     10     for mod in list(sys.modules):
     11         if mod == package or mod.startswith(package + '.'):
---> 12             sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
     13 
     14 # Kinda cool, though, right?

KeyError: 'pip._vendor.urllib3.contrib'

The text was updated successfully, but these errors were encountered:

sernst · 2018-03-17T15:12:15Z

I'm seeing the same thing with pip 9.0.2 while running on the python:3.6 docker container

pip version: 9.0.2
python version: 3.6.4
operating system: docker/linux

I've been able to narrow it down to occurring when the requests library is imported before the pip library. Here's an example:

docker-user:~# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import pip
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/pip/__init__.py", line 45, in <module>
    from pip.vcs import git, mercurial, subversion, bazaar  # noqa
  File "/usr/local/lib/python3.6/site-packages/pip/vcs/mercurial.py", line 9, in <module>
    from pip.download import path_to_url
  File "/usr/local/lib/python3.6/site-packages/pip/download.py", line 40, in <module>
    from pip._vendor import requests, six
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/__init__.py", line 98, in <module>
    from . import packages
  File "/usr/local/lib/python3.6/site-packages/pip/_vendor/requests/packages.py", line 12, in <module>
    sys.modules['pip._vendor.requests.packages.' + mod] = sys.modules["pip._vendor." + mod]
KeyError: 'pip._vendor.urllib3.contrib'
>>> exit()
docker-user~# pip --version
pip 9.0.2 from /usr/local/lib/python3.6/site-packages (python 3.6)

The reverse works fine:

docker-user:~# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import requests
>>> print(pip)
<module 'pip' from '/usr/local/lib/python3.6/site-packages/pip/__init__.py'>

dstufft · 2018-03-17T15:13:39Z

Are you importing pip in your code? Or is there a pip command you're running to trigger this?

sernst · 2018-03-17T15:24:32Z

In my case, I'm importing pip in my code to use pip.main().

dstufft · 2018-03-17T15:25:35Z

Does it happen on the master branch as well?

dstufft · 2018-03-17T15:28:27Z

(If you're using pip.main() that's been renamed to pip._internal.main() in master as we don't have any supported APIs).

sernst · 2018-03-17T15:35:46Z

Was just checking that. It looks like master works fine:

docker-user:~/libraries# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> import pip
>>> print(pip)
<module 'pip' from '/root/libraries/src/pip/__init__.py'>
>>> print(pip.__version__)
10.0.0.dev0

sernst · 2018-03-17T15:36:05Z

(good to know about the pip.main() refactor)

dstufft · 2018-03-17T15:36:44Z

You'll need to import pip._internal, there's nothing in pip/__init__.py except the __version__.

sernst · 2018-03-17T15:38:35Z

Ah, yep. Seeing that now. Still seems to work:

docker-user:~/libraries# python
Python 3.6.4 (default, Mar 14 2018, 17:49:05)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from pip import _internal
>>> print(_internal)
<module 'pip._internal' from '/root/libraries/src/pip/_internal/__init__.py'>
>>> import pip
>>> print(pip.__version__)
10.0.0.dev0

dstufft · 2018-03-17T15:45:24Z

Alright. I'm inclined to say that that you'll want to either re-arrange your imports, or pin to 9.0.1. While I recognize that it sucks that it broke, we don't really support any usage of pip where you're importing pip itself (hence the refactor to move everything into pip._internal to make this crystal clear, since a lot of people were confused by this). In addition, we're planning on releasing in the upcoming weeks (see the announcement) which doesn't appear to have this problem.

This is largely because 9.0.x has drifted enough that cutting 9.0.2 was an ~8 hour process, and I'd much rather not have to do that again, particularly for an unsupported use of pip. If one of the other pip developers feel strongly about it, they're welcome to do it though.

Sorry again!

sernst · 2018-03-17T16:13:41Z

Yep, I can do that to solve my case. 👍

Understandable that it's not officially supported, but there's definitely usage of it out there. GitHub code search won't allow for using . so this search isn't perfect:

https://github.com/search?l=Python&q=import+pip+main%28&type=Code&utf8=%E2%9C%93

but it returns ~850k hits of import pip + main( on public github projects.

An anecdotal pattern I notice from a quick look through them, is that many are installing data packages. Not surprising since there are many data environments (notebook deployments especially) that obfuscate access to the host environment. This is serving as an in-python way of managing dependencies that can be shared in such contexts.

I'm not sure if there's any interest in maintaining the pip.main() as a passthrough command execution wrapper, but that could potentially mitigate the pain of upgrading.

Just a thought though, I've already refactored to resolve my issue.

dstufft · 2018-03-17T16:17:50Z

I'm not sure if there's any interest in maintaining the pip.main() as a passthrough command execution wrapper, but that could potentially mitigate the pain of upgrading.

I think this would make an excellent issue to discuss the relative merits of doing that (or not doing it).

nchammas · 2018-03-19T14:51:47Z

FWIW I'm seeing something very similar just by calling pip install on some requirements:

++ pip install -U pip
Collecting pip
  Using cached pip-9.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-9.0.2
++ pip install -r requirements.pip
Collecting jellyfish==0.5.6 (from -r requirements.pip (line 1))
  Using cached jellyfish-0.5.6.tar.gz
Collecting numpy==1.13.1 (from -r requirements.pip (line 2))
  Using cached numpy-1.13.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting vertica-python==0.7.3 (from -r requirements.pip (line 3))
Exception:
Traceback (most recent call last):
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/index.py", line 792, in get_page
    "Cache-Control": "max-age=600",
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/adapter.py", line 37, in send
    cached_response = self.controller.cached_request(request)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/controller.py", line 111, in cached_request
    resp = self.serializer.loads(request, cache_data)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/serialize.py", line 114, in loads
    return getattr(self, "_loads_v{0}".format(ver))(request, data)
  File "/hadoop/jenkins_home/workspace/sample-pyspark-application/venv/lib/python3.5/site-packages/pip/_vendor/cachecontrol/serialize.py", line 168, in _loads_v1
    cached = pickle.loads(data)
ImportError: No module named 'pip._vendor.requests.packages.urllib3'; 'pip._vendor.requests.packages' is not a package

Strangely, I can avoid the error by calling pip install on my requirements with --no-cache-dir. So perhaps in my case there is some bad interaction involving pip 9.0.2 and cached packages?

Importing pip 9.0.2 from the tests causes a crazy build failure. Apparently importing pip the way we do has never been supported and we should stop doing it. However, this stops the bleeding until we can figure out what to do correctly. Alternately, it's possible that when pip 10 comes out, the problem will go away by itself and we can continue doing the unsupported thing that we were doing. See also pypa/pip#5079 (comment).

nchammas · 2018-03-19T15:31:48Z

Alright, well I was able to fix my problem by deleting the system pip cache. Now my jobs run as they were before, without needing to pin pip to 9.0.1 or specifying --no-cache-dir.

Dunno if this is related at all to what people are seeing here or on #5081, but noting my experience here nonetheless in case it helps someone.

Pip does a bunch of magic vendoring of packages for its internal use. In pip 9.0.2, the logic of this changed in a way such that it is sometimes necessary for pip to be imported *before* anything else. There are some details on this in this thread: pypa/pip#5079 Ultimately, we will want to stop importing pip in our code. This is just a temporary fix so that things work in the short term.

- policy-handler uses dns based discovery of deployment-handler - driven by config - new data structure for deploy_handler section of config -- changed from string "deployment_handler" in 2.3.1 to structure in 2.4.0 deploy_handler : # name of deployment-handler service # used by policy-handler for logging target_entity : "deployment_handler" # url of the deployment-handler service # for policy-handler to direct the policy-updates to # - expecting dns to resolve the name # deployment_handler to ip address url : "http://deployment_handler:8188" - logic is backwards compatible with 2.3.1 format - removed import pip from audit -- import pip broken in pip 9.0.2 (2018-03-19) -- import pip conflicts with requests -- pip API is not officially supported -- see links for more pypa/pip#5079 pypa/pip#5081 Change-Id: Ifcaba6cfd714f3099ab7a25fe979a3696a6460fc Signed-off-by: Alex Shatov <alexs@att.com> Issue-ID: DCAEGEN2-404

jberends · 2018-03-20T18:24:49Z

👍 I experience the problem inside the build of docker containers based on python:2.7-slim, python:3.5-slim etc. So I cannot do finegrained control of --no-cache-dirs. 9.0.2 is already installed. I am calling pip.main() from inside a python script.

Currently there is a reported problem in pip library [1]. To avoid this problem, the pip version should be enforced during the installation. [1]pypa/pip#5079 JIRA: YARDSTICK-1087 Change-Id: I2ee28749f146cd3b0e1bf4ed92b51cbe6db8b2f3 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

* Update docs/submodules/yardstick from branch 'master' - Merge "Enforce pip version during installation" - Enforce pip version during installation Currently there is a reported problem in pip library [1]. To avoid this problem, the pip version should be enforced during the installation. [1]pypa/pip#5079 JIRA: YARDSTICK-1087 Change-Id: I2ee28749f146cd3b0e1bf4ed92b51cbe6db8b2f3 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

Previously, we used internal pip code, which was changing in point releases, causing issues. We actually don't need pip to do this, we can instead ask the installed python's pkg_resources directly for the same info. Related issues: pypa/pip#5079 pypa/pip#5080 pypa/pip#5081

* Update docs/submodules/yardstick from branch 'master' - Merge "Re enable pip package unit tests" - Re enable pip package unit tests In [1] the pip unit test cases were removed, because of a problem in pip3 library. Currently there is a reported error in pip 9.0.2 version, in _vendor.urllib3 library [2]. This error is solved in [3]. [1]I43b1edc23cebe0db97b6b56a551acd376986dfea [2]pypa/pip#5079 [3]I2ee28749f146cd3b0e1bf4ed92b51cbe6db8b2f3 JIRA: YARDSTICK-1087 Change-Id: Id4f0e1accf3240b5f27d3698323e679a27f63f87 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

In [1] the pip unit test cases were removed, because of a problem in pip3 library. Currently there is a reported error in pip 9.0.2 version, in _vendor.urllib3 library [2]. This error is solved in [3]. [1]I43b1edc23cebe0db97b6b56a551acd376986dfea [2]pypa/pip#5079 [3]I2ee28749f146cd3b0e1bf4ed92b51cbe6db8b2f3 JIRA: YARDSTICK-1087 Change-Id: Id4f0e1accf3240b5f27d3698323e679a27f63f87 Signed-off-by: Rodolfo Alonso Hernandez <rodolfo.alonso.hernandez@intel.com>

hemangjoshi37a · 2018-12-15T09:19:13Z

How to do it in windows??

This is a workaround for `pip` 9.0.2, which is the version that runs on Heroku. For some reason, importing `isort` import imports `requests`, and if we import `requests` before/without importing `pip`, we get a `KeyError`. We only need for this import to succeed locally and on CI (basically, anywhere where we want to run our test suite), so it's ok for this to fail on production. For more info: pypa/pip#5079 (comment)

* Gracefully ignore KeyError on isort import This is a workaround for `pip` 9.0.2, which is the version that runs on Heroku. For some reason, importing `isort` import imports `requests`, and if we import `requests` before/without importing `pip`, we get a `KeyError`. We only need for this import to succeed locally and on CI (basically, anywhere where we want to run our test suite), so it's ok for this to fail on production. For more info: pypa/pip#5079 (comment) * Bump version number

lock · 2019-05-31T04:19:21Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

dstufft closed this as completed Mar 17, 2018

sernst mentioned this issue Mar 17, 2018

Preserve pip.main() in some fashion #5080

Closed

alex mentioned this issue Mar 17, 2018

Error with import pip in pip 9.0.2 #5081

Closed

rstein mentioned this issue Mar 17, 2018

Installer exits with KeyError: 'pip._vendor.urllib3.contrib' modoboa/modoboa-installer#188

Closed

mdcclv mentioned this issue Mar 18, 2018

[v1.x] Backport 2.x's avoidance of 'import pip' explosion/spaCy#2110

Merged

3 tasks

jantman mentioned this issue Mar 18, 2018

ImportError when running with pip==9.0.2 jantman/versionfinder#5

Closed

jantman added a commit to jantman/versionfinder that referenced this issue Mar 18, 2018

pip import fixup for pip10 and pypa/pip#5079

61632c1

jantman mentioned this issue Mar 18, 2018

pip 9.0.2 and 10 fixes jantman/versionfinder#6

Merged

playpauseandstop mentioned this issue Mar 18, 2018

poetry doesn't work with pip==9.0.2 python-poetry/poetry#17

Closed

maisim mentioned this issue Mar 19, 2018

"deploy instance" failed modoboa/modoboa#1437

Closed

kvikshaug mentioned this issue Mar 20, 2018

Drop usage of unsupported pip api opencobra/cobrapy#680

Closed

foosel mentioned this issue Mar 20, 2018

Error message ends with %(error)s instead of actual error message OctoPrint/OctoPrint#2499

Closed

drmattyg mentioned this issue Mar 20, 2018

import nbconvert or nbformat causes importing pip 9.0.2 to fail jupyter/nbconvert#783

Closed

higs4281 mentioned this issue Mar 21, 2018

Catch misleading ImportError #5098

Closed

CRiva mentioned this issue Mar 21, 2018

Unable to import apicontrollers in python 3.5.2 AuthorizeNet/sdk-python#96

Closed

bluespider42 mentioned this issue Mar 23, 2018

KeyError: 'pip._vendor.urllib3.contrib' Python3.6 pip 9.0.2 pyupio/pyup#285

Closed

ke1337 mentioned this issue Mar 28, 2018

KeyError: 'pip._vendor.urllib3.contrib' microsoft/CNTK#3047

Open

JPeterMugaas mentioned this issue Sep 11, 2018

python-pip - retry comit msys2/MINGW-packages#4379

Merged

brainix mentioned this issue Feb 26, 2019

Gracefully ignore isort import keyerror brainix/pottery#110

Merged

JeremyLoy mentioned this issue Feb 28, 2019

pip 9.0.2 can break Pipenv builds - need to change pip version heroku/heroku-buildpack-python#804

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label May 31, 2019

lock bot locked as resolved and limited conversation to collaborators May 31, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip 9.0.2 KeyError: 'pip._vendor.urllib3.contrib' #5079

pip 9.0.2 KeyError: 'pip._vendor.urllib3.contrib' #5079

vhaldemar commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

nchammas commented Mar 19, 2018

nchammas commented Mar 19, 2018

jberends commented Mar 20, 2018 •

edited

Loading

hemangjoshi37a commented Dec 15, 2018

lock bot commented May 31, 2019

pip 9.0.2 KeyError: 'pip._vendor.urllib3.contrib' #5079

pip 9.0.2 KeyError: 'pip._vendor.urllib3.contrib' #5079

Comments

vhaldemar commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

sernst commented Mar 17, 2018

dstufft commented Mar 17, 2018

nchammas commented Mar 19, 2018

nchammas commented Mar 19, 2018

jberends commented Mar 20, 2018 • edited Loading

hemangjoshi37a commented Dec 15, 2018

lock bot commented May 31, 2019

jberends commented Mar 20, 2018 •

edited

Loading