New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip 9 breaks the forward compatibility of its cache for chunked responses #4078

Closed
xavfernandez opened this Issue Nov 3, 2016 · 0 comments

Comments

Projects
None yet
1 participant
@xavfernandez
Contributor

xavfernandez commented Nov 3, 2016

Symptoms:

$ pip -V
pip 9.0.0 from /home/xfernandez/.virtualenvs/tmp-3e2d3e3998def93/lib/python3.5/site-packages (python 3.5)
$ pip install pip==8.1.2
Collecting pip==8.1.2
  Using cached https://pypi.private.fr/root/pypi/+f/057/0520434c5b600/pip-8.1.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 9.0.0
    Uninstalling pip-9.0.0:
      Successfully uninstalled pip-9.0.0
Successfully installed pip-8.1.2
$ pip install pip==9
Collecting pip==9
  Could not find a version that satisfies the requirement pip==9 (from versions: )
No matching distribution found for pip==9
$ pip install pip==9 -v
....
Collecting pip==9
  1 location(s) to search for versions of pip:
  * https://user:password@pypi.private.fr/jenkins/jenkins/+simple/pip/
  Getting page https://user:password@pypi.private.fr/jenkins/jenkins/+simple/pip/
  Looking up "https://pypi.private.fr/jenkins/jenkins/+simple/pip/" in the cache
  Current age based on date: 44
  Freshness lifetime from expires: 0
  Freshness lifetime from request max-age: 600
  The response is "fresh", returning cached response
  600 > 44
  Analyzing links from page https://pypi.private.fr/jenkins/jenkins/+simple/pip/
  Could not find a version that satisfies the requirement pip==9 (from versions: )
Cleaning up...
...
$ rm -rf ~/.cache/pip/http/
$ pip install pip==9
Collecting pip==9
  Downloading https://pypi.private.fr/root/pypi/+f/55f/554c8be55cba2/pip-9.0.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 1.3MB/s
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-9.0.0

After investigating, the issue comes from ionrock/cachecontrol@e170553 that was included in pip 9.0.
It adds the possibility to cache chunked responses so pip 9 now does that (which is good).

Unfortunately if you downgrade pip to a previous version, it will try to use the cached chunked response while lacking the necessary code to do that (specifically: https://github.com/ionrock/cachecontrol/blob/e17055379e2c6c373f65fcf9eee09b7b828b202d/cachecontrol/serialize.py#L137-L141).

pip then uses the cached response and pass the Transfer-Encoding: chunked header to urllib3 that ends up crashing here: https://github.com/pypa/pip/blob/8.1.2/pip/_vendor/requests/packages/urllib3/response.py#L444 with AttributeError: '_io.BytesIO' object has no attribute 'fp'.

The exception bubbles up here: https://github.com/pypa/pip/blob/8.1.2/pip/_vendor/requests/models.py#L741 and we end up with an empty response: https://github.com/pypa/pip/blob/8.1.2/pip/_vendor/requests/models.py#L744 which means that pip sees an empty index page without any version available. Hence the Could not find a version that satisfies the requirement pip==9 (from versions: ) error message.

As a matter of fact, adding https://github.com/ionrock/cachecontrol/blob/e17055379e2c6c373f65fcf9eee09b7b828b202d/cachecontrol/serialize.py#L137-L141 to pip 8.1.2 allows pip 8.1.2 to deal with pip 9.0 cache.

This whole mess only happens with chunked responses so an easy workaround if you share your cache between several pip versions is to avoid chunked responses.

PyPI does not use them. If your private PyPI is using them, you can certainly disable them in your web server.
With nginx, chunked_transfer_encoding off; does the trick.

@xavfernandez xavfernandez added this to the 9.0 milestone Nov 3, 2016

xavfernandez added a commit to xavfernandez/pip that referenced this issue Nov 3, 2016

Add warning for forward compatibility breakage
in the cache for chunked responses (cf issue pypa#4078)

dstufft added a commit that referenced this issue Nov 3, 2016

Add warning for forward compatibility breakage (#4079)
in the cache for chunked responses (cf issue #4078)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment