pip 10: "pip list" warns "Cache entry deserialization failed, entry ignored"

[kkdd]

Hello,
pip v.10 repeatedly warns "Cache entry deserialization failed, entry ignored":

$ uname
Darwin
$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.13.4
BuildVersion:	17E199
$ brew info python@2
python@2: stable 2.7.14 (bottled), devel 2.7.15rc1, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@2/2.7.14_3 (4,603 files, 81.8MB) *
  Poured from bottle on 2018-03-10 at 22:25:36
 :
 :
$ brew info python3
python: stable 3.6.5 (bottled), devel 3.7.0b3, HEAD
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python/3.6.5 (4,768 files, 100.2MB) *
  Poured from bottle on 2018-03-30 at 20:43:21
 :
 :
$ pip2 -V
pip 10.0.0 from /usr/local/lib/python2.7/site-packages/pip (python 2.7)
$ pip3 -V
pip 10.0.0 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)
$ pip2 list --outdated
Package  Version Latest      Type 
-------- ------- ----------- -----
protobuf 3.5.1   3.5.2.post1 wheel
$ pip3 list --outdated
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Cache entry deserialization failed, entry ignored
Package  Version Latest      Type 
-------- ------- ----------- -----
protobuf 3.5.1   3.5.2.post1 wheel
$ 

[japagetw]

I have also noted the same problem for the 64-bit versions of Windows 7 and 10 Professional. I see the problem only for Python 3.5 and not Python 2.7 or Python 3.6 (I have separate installations of all 3 versions). Adding --format legacy or --no-cache-dir makes the problem go away.

[jackton1]

Temporary Fix upgrade pip3 using.

curl https://bootstrap.pypa.io/get-pip.py | python3

Instead of pip install -U pip

For pip2 pip2 install --upgrade pip

[pradyunsg]

It seems like you have messed up your pip cache. It's stored in ~/.cache/pip.

If you delete that directory, this issue should be fixed.

[kkdd]

Thank you. This phenomenon reproduces. I have confirmed that pip3 list --no-cache-dir --outdated works without warnings.

[japagetw]

I suspect that the problem is that cache entries created in Python 2 cannot be deserialized in Python 3. If I use the --cache-dir option to force different caches for Python 2 and Python 3, the problem goes away.

The error message is generated from C:\Python36\Lib\site-packages\pip_vendor\cachecontrol\controller.py line 139. The proximate cause is a failure of Serializer.loads() in C:\Python36\Lib\site-packages\pip_vendor\cachecontrol\serialize.py line 75.

[kkdd]

Thank you. Your analysis sounds reasonable.

[xflr6]

@pradyunsg: The problem is not fixed by deleting the cache dir (see the comment of @japagetw).

AFAIU, using --no-cache-dir on Python 3 is only a temporary workaround for this (i.e., it is a bug).

Can we make the shared cache dir for Python 2 and 3 work again (or is there a reason why the same version of pip >= 10 needs to use a different cache format on PY2 vs. PY3)?

[ssbarnea]

This seems like a serious bug and one quick workaround should be to assure that cached entries filenames contain a serialisation version part inside them.

[HakShak]

The fact that this is also on stderr is quite annoying.

[AronT-TLV]

Deleting directory in .cache and creating 2 new ones pip2 and pip3 in that directory, solved the issue for me on one of my Macs temporarily, but not on another. Reinstalling pip3 via curl did not help. In fact, the problem appears inconsistently and mostly, but not exclusively, with pip3 not pip2.

[pradyunsg]

This is likely due to a bug in the caching pipeline -- there's a bit of history with msgpack doing stuff incorrectly around bytes.

If someone can take a look into this, before one of the maintainers finds the time to, that would be cool. :)

[ssbarnea]

Is anyone planning to do something about this newly introduced bug? pip should be updated ASAP to avoid clashing incompatible caches.

The workarounds mentioned here are not really applicable in all cases and are likely to cause more problems than sorting. Adding a versioning part in the cache entry naming would address this issue once for all, even if the cache format would change again.

Potentially every python package developer would encounter this issue if they use tox to test with multiple python versions. Please don't ask them to isolate the caches for each tox target, this defeats the purpose of having a cache.

[giganut]

I'm having this problem as well.

[cosmiccamel]

Im having the same problem as well

[kamikaze]

what happens?? too many bugs and problems since 9.0.1

[ssbarnea]

@HakShak The fact stderr is used is a good thing. POSIX specification documents and requires that all logging messages would go to stderr and not stdout, including errors, warnings, info, debug. Only the normal program output should go to stdin. This usually mean, for most tools: computer parsable output. For example on a package manager stdout should contain only installed/removed/upgraded packages, without the boilerplate which should go to stderr. I remember fixing few non compliant tools over the years. The output stream is not the issue here.

On the other hand, I just observed that this bug was added to the 18.0 milestone, not sure how to take this, as current version is 10.0. Is like a very long time in the future, or maybe I don't know about a plan to boost pip version to match current year?

[pradyunsg]

Is like a very long time in the future, or maybe I don't know about a plan to boost pip version to match current year?

We switched to CalVer (#5324), 18.0 is going to be the next release of pip, likely next month.

[pradyunsg]

I found that on Python 2, pip is able to use a cache populated by a Python 3 run but not vice versa.

A response cached on Py2 results in the cache storing a header that loads on Py3 as bytes instead of str. This causes the deserialization to fail. Working on a PR to fix it upstream.

[pradyunsg]

Upstream PR: psf/cachecontrol#190

[hugovk]

Upstream PR psf/cachecontrol#190 is merged and released in CacheControl 0.12.5.

[pradyunsg]

pip will absorb that change when we update our vendored dependencies in preparation for the next release. :)

[Mavericks334]

I am having the same problem.

This is the error i get

Cache entry deserialization failed, entry ignored
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))': /simple/pip/
Requirement already up-to-date: pip in c:\programdata\anaconda3\lib\site-packages
You are using pip version 9.0.3, however version 18.0 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

How do i avoid this. Due to this i am unable to install any other packages since it is asking me to upgrade pip.

[etanot]

It seems like you have messed up your pip cache. It's stored in ~/.cache/pip.

If you delete that directory, this issue should be fixed.

I'm more interested in, what is cache entry deserialization?

[pradyunsg]

#5250 (comment) elaborates on why it failed.

In essence, the error occurs due to a mismatch when converting an http request into a string (serialization). This error/warning shows up when trying to convert that string into an http request again (deserialization).

[IshikuUltra]

It seems like you have messed up your pip cache. It's stored in ~/.cache/pip.

If you delete that directory, this issue should be fixed.

Thank you so much.

[Gaopeng-Bai]

Easy to resolve this problem, run app or terminal as administrator

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.