Unable to clean http cache with pip

[eamars]

Environment

• pip version: pip 20.2.3
• Python version: Python 2.7 (32bit)
• OS: Windows 10 (64bit)

Description

We noticed pip will cache the http response for future installation. However if the cached http response includes corruped data (network issue for example), installing the same package will result in pip cache error as follow:

(venv2) H:\>pip install pyside
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pyside
  Using cached PySide-1.2.4-cp27-none-win32.whl (41.0 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pyside from https://files.pythonhosted.org/packages/39/78/2b5fcd3b4ff4fc891f1c3a8bee09616d59fa6d644b0dc6252d46d8fbb423/PySide-1.2.4-cp27-none-win32.whl#sha256=104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908:
        Expected sha256 104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908
             Got        9c9bdae0c8b07e8e834b3bd9f35ab93f3ddbae8365047b1596553eb70e996427

The symption of this issue has already been reported in pypi/warehouse#8330.

Expected behavior

If the error is caused by the data corruption in http response cache then pip should invalidate the cache and try to re-download for best attempt.

Currently pip cache purge will not clear http response cache. The only solution to this case is to remove http directory from the file system forcing pip to rebuild http cache.

How to Reproduce

1. Create a Python 2 virtual environment
2. Download the example corruped file https://mega.nz/file/WsthyLTS#AWD7NmS-w9B62Q3Y8Lb4SvCalqCb1d83a5FniKPmFqY
3. Overwrite http folder in C:\Users<account name>\AppData\Local\pip\cache\http
4. Uninstall pyside and install pyside by pip install pyside

Output

(venv2) H:\>pip install pyside
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Collecting pyside
  Using cached PySide-1.2.4-cp27-none-win32.whl (41.0 MB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    pyside from https://files.pythonhosted.org/packages/39/78/2b5fcd3b4ff4fc891f1c3a8bee09616d59fa6d644b0dc6252d46d8fbb423/PySide-1.2.4-cp27-none-win32.whl#sha256=104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908:
        Expected sha256 104c0f3aee597e92c43b60f31205cbbd57a4307fea0fae4bbaeeb58b61878908
             Got        9c9bdae0c8b07e8e834b3bd9f35ab93f3ddbae8365047b1596553eb70e996427

[hugovk]

Would it make sense for the pip cache commands to include the http directory too?

Looking at the cache subcommands (dir, info, list, purge, remove):

$ # No change needed:
$ pip cache dir
/Users/hugo/Library/Caches/pip

$ # Now:
$ pip cache info
Location: /Users/hugo/Library/Caches/pip/wheels
Size: 67.1 MB
Number of wheels: 139

$ # Next:
$ pip cache info
http location: /Users/hugo/Library/Caches/pip/http
http size: 1202.4 MB
Wheels location: /Users/hugo/Library/Caches/pip/wheels
Wheels size: 67.1 MB
Number of wheels: 139

$ # Now (and there's probably no point listing any `http` files?):
$ pip cache list
Cache contents:

 - MacFSEvents-0.8.1-cp38-cp38-macosx_10_9_x86_64.whl (13 kB)
 - Pillow-4.3.0-cp38-cp38-macosx_10_9_x86_64.whl (491 kB)
...
 - ujson-2.0.3-cp38-cp38-macosx_10_9_x86_64.whl (45 kB)
 - wrapt-1.12.1-cp38-cp38-macosx_10_9_x86_64.whl (32 kB)

$ # Now:
$ pip cache purge
Files removed: 139
$ # Next:
$ pip cache purge
Files removed: 2245

$ # Now (and there's probably no point removing matched `http` files?):
$ pip cache remove "tiny*"
Files removed: 2

[hugovk]

Please see PR #8910.

[eamars]

Thanks @hugovk for pointing out the one of the potential fix!.

However, the other issue I mentioned in my post was the pip kept using the corruped HTTP cache. Would it make sense to invalidate the cache if corruption is detected?

[hugovk]

Would it make sense to invalidate the cache if corruption is detected?

Sounds reasonable (and whether to invalidate part or the whole cache) but I'll leave it to someone who's more familiar with pip's caching to comment further.

[pradyunsg]

@eamars how would pip detect corruption, given that we don't check the hashes until after it is placed in the cache?

Honestly, I feel like hash checking by default in pip (ala TUF on PyPI - there's PEP 458 and stuff written down, can't find the GitHub issue for it) should basically make it impossible to have an invalid/corrupted wheel making it into the cache, so... I'm not too worried about fixing this ASAP given that there's probably something on the roadmap already?