Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed behavior with --extra-index-url with pip 9 #4195

Closed
hjc opened this issue Dec 22, 2016 · 8 comments

Comments

Projects
None yet
7 participants
@hjc
Copy link

commented Dec 22, 2016

  • Pip version: 9.0.1
  • Python version: 3.5.2+
  • Operating System: Debian Stretch (testing)

Description:

We have a private internal package we use for our projects that is hosted on our own internal server. That packaging server only has our private packages and is not a full PyPI mirror, because it's not what we need. In the past, I have been able to run:

pip install -r requirements.txt --extra-index-url http://pip.croscon.com:8088/simple/ --trusted-host pip.croscon.com

and it works fine. However, now when I run this command, I get an error from pip saying it can't find a specific package on pip.croscon.com. This package is not our internal package and is one we normally pull from PyPI. This critically errors and fails. Going back to pip 8 fixed this.

What I've run:

(.env) hayden@beard-machine ~/g/zvm-demo> pip --version
pip 9.0.1 from /home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages (python 3.5)
(.env) hayden@beard-machine ~/g/zvm-demo> pip install -r requirements.txt  \
                                            --extra-index-url http://pip.croscon.com:8088/simple/  \
                                            --trusted-host pip.croscon.com
Collecting Flask-Celery-Helper==1.1.0 (from -r requirements.txt (line 1))
Exception:
Traceback (most recent call last):
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/home/hayden/gits/zvm-demo/.env/share/python-wheels/requests-2.11.1-py2.py3-none-any.whl/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://pip.croscon.com:8088/simple/flask-celery-helper/
# now swap to pip 8
(.env) hayden@beard-machine ~/g/zvm-demo> pip install -U "pip>=8.0.0,<9.0.0"
Collecting pip<9.0.0,>=8.0.0
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 895kB/s 
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-8.1.2
(.env) hayden@beard-machine ~/g/zvm-demo> pip install -r requirements.txt  \
                                            --extra-index-url http://pip.croscon.com:8088/simple/  \
                                            --trusted-host pip.croscon.com
  ... omitted for brevity; but it was pip working as expected ...
Successfully installed Flask-0.12 Flask-Celery-Helper-1.1.0 Flask-Classful-0.13.1 Flask-Login-0.4.0 Flask-Marshmallow-0.7.0 Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.11.11 amqp-1.4.9 anyjson-0.3.3 arrow-0.10.0 billiard-3.3.0.23 blinker-1.4 celery-3.1.25 click-6.6 eventlet-0.19.0 fleaker-0.1.4 google-auth-0.5.0 google-auth-httplib2-0.0.2 google-cloud-core-0.22.1 google-cloud-storage-0.22.0 googleapis-common-protos-1.5.0 greenlet-0.4.11 gunicorn-19.6.0 httplib2-0.9.2 invoke-0.12.1 itsdangerous-0.24 kombu-3.0.37 marshmallow-2.10.5 marshmallow-jsonschema-0.3.0 oauth2client-4.0.0 pendulum-0.7.0 protobuf-3.1.0.post1 pyasn1-0.1.9 pyasn1-modules-0.0.8 python-dateutil-2.6.0 pytz-2016.10 pytzdata-2016.10 redis-2.10.5 rsa-3.4.2 six-1.10.0 tzlocal-1.3
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
@hjc

This comment has been minimized.

Copy link
Author

commented Jan 26, 2017

Hello?

Any response? Hit this again today which reminded me of the problem at hand.

@hjc

This comment has been minimized.

Copy link
Author

commented Jan 26, 2017

Interestingly enough, this is an issue with virtualenv. The pip it installs is just broken by default. Downgrading to pip 8 and then upgrading back to pip 9 fixes it.

Virtualenv version is: 15.1.0.

@hjc hjc closed this Jan 26, 2017

@teej-minted

This comment has been minimized.

Copy link

commented Sep 26, 2017

Just hit this exact same issue (private repo, pip install fails saying it cant find the package). We followed the exact same procedure (downgrade to ~8, upgrade back to 9) and it worked. We are all shaking our heads at this.

@dstufft

This comment has been minimized.

Copy link
Member

commented Sep 26, 2017

Are you using virtualenv that came with Debian? They patch the pip they install with that and the upgrade/downgrade dance ends up forcing you to use the unpatched version from PyPI. It could be an issue with the patching done by Debian.

@arthurdarcet

This comment has been minimized.

Copy link

commented Nov 15, 2017

This is definitely a bug in the debian patch for pip ; it was reported here https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837764 a long time ago but went dead. It is still happening today.

@cboylan

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

I don't actually think this is completely a downstream bug. The issue is related to how debian unvendors vendored libs, but we trip over that because the exception is pip._vendor.requests.exceptions.HTTPError not pip._vendor.requests.HTTPError.

I'll get a PR up shortly that seems to fix this for debian (and also happens to be more like the other exception handling in index.py).

cboylan added a commit to cboylan/pip that referenced this issue Mar 29, 2019

Properly catch requests' HTTPError in index.py
This resolves issue pypa#4195.

In index.py's index retrieval routine we were catching
requests.HTTPError to log and ignore 404s and other similar HTTP server
errors when pulling from (extra-)index-urls. Unfortunately, the actual
path to that exception is requests.exceptions.HTTPError and the alias we
were using does not work when pip is installed with unvendored libs as
with the debian packaged pip.

Thankfully the fix is simple. Import and use
requests.exceptions.HTTPError. This comes with the added bonus of
fitting in with the existing handling for RetryError and SSLError. With
this change in place upstream pip and downstream packaged pip should
both catch this exception properly.

Note: I've not added any tests cases as I'm unsure how to test the
distro packaging case within pip's testsuite. However, the existing test
suite should hopefully cover that this isn't a regression and I've
manually confirmed that this works with a hacked up debian package
install. Also this is how we handle RetryError and SSLError.

cboylan added a commit to cboylan/pip that referenced this issue Mar 31, 2019

Properly catch requests' HTTPError in index.py
This resolves issue pypa#4195.

In index.py's index retrieval routine we were catching
requests.HTTPError to log and ignore 404s and other similar HTTP server
errors when pulling from (extra-)index-urls. Unfortunately, the actual
path to that exception is requests.exceptions.HTTPError and the alias we
were using does not work when pip is installed with unvendored libs as
with the debian packaged pip.

Thankfully the fix is simple. Import and use
requests.exceptions.HTTPError. This comes with the added bonus of
fitting in with the existing handling for RetryError and SSLError. With
this change in place upstream pip and downstream packaged pip should
both catch this exception properly.

Note: I've not added any tests cases as I'm unsure how to test the
distro packaging case within pip's testsuite. However, the existing test
suite should hopefully cover that this isn't a regression and I've
manually confirmed that this works with a hacked up debian package
install. Also this is how we handle RetryError and SSLError.

@cjerdonek cjerdonek added the type: bug label Mar 31, 2019

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Mar 31, 2019

Reopening as this was found in #6367 to be an issue with pip’s DEBUNDLED section. (Thanks, @cboylan, for your contribution here!)

@cjerdonek

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

Addressed by PR #6373 / 5af0bc3.

@cjerdonek cjerdonek closed this Apr 3, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.