virtualenv creation may hang or raise oserror due to pip cache invalidation

[ssbarnea]

It seems that something may happen to pip cache(~/.cache/pip) that could render virtualenv unusable, it will hang for minutes and eventually fail with a meaningless error.

One message that worth mentioning is Cache entry deserialization failed, entry ignored but I was not able to find exactly what it means.

Note that I did kept a copy of the broken ~/.cache/pip so we can perform additional tests on it in order identify which kind of issue we had and how to avoid/detect/repair it.

I already checked and all file permissions are ok, also apparently there are no spaces in any filenames.
A full list of files is available at https://gist.github.com/ssbarnea/5d6515768dadcb86713e06349b4fbcb8

This happened on CI so we really need to find a way to avoid this from recurring.

$ virtualenv -vvv ~/xx
...
Installing setuptools, pip, wheel...
  Running command /home/rhos-ci/xx/bin/python - setuptools pip wheel
  Collecting setuptools
    Using cached setuptools-36.8.0-py2.py3-none-any.whl
  Collecting pip
  Collecting wheel
    Cache entry deserialization failed, entry ignored
^C...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 769, in call_subprocess
    line = stdout.readline()
KeyboardInterrupt

If I didn't press CTRL-C, it would naturally fail after a big number of minutes, with

Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/usr/lib/python2.7/site-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /home/rhos-ci/xx/bin/python - setuptools pip wheel failed with error code -15

[pombredanne]

Which OS and exact Python 2 version do you run? which virtualenv and pip and wheel versions?
Somehow your cache may be damaged. Does this happen again if you clear it?

[ssbarnea]

I was able to narrow down the issue and also to find a solution. It seems that httplib2-0.9.2 is broken but httplib2-0.10.3 seems to be working fine.

Sounds like a call to add some minimal dependencies for httplib2 when on python2.7

[pradyunsg]

I think you're hitting pypa/pip#3532.

Deleting the cache is a known workaround. Or... you could find the relevant lock file that lockfile creates and delete that. ;)

I think it'll be http/d/9/f/5/0/d9f5007c5cc7fe7a953d81df1938d0a96573b4be7f6b4aea55ab2559.lock.

I was not able to find exactly what it means.

It's from cachecontrol -- a dependency vendored in pip who's spewing that message. It's because some entry in the cache is messed up; probably because of a KeyboardInterrupt or something.

I think pip should start removing invalid entries from the cache but then it just spews this message and ignores them so that's not a cause of concern IIRC.

[ssbarnea]

@pradyunsg There is a serious cause of concern: without upgrading httplib2 virtualenv will be totally broken when this occurs. I don't really care about this message if is benign, but is not, is a clear indication that the bug was triggered.

My current workaround was to add sudo pip install --upgrade-strategy only-if-needed 'httplib2>=0.10.3' to configuration management scripts, with a note pointing to this issue. Still I will try to assure that this is fixed upstream as I don't fancy those workarounds.... we all know how nice is to drawn in them ;)

[pradyunsg]

It's a issue caused by the specific cache on that machine.

Is this on Travis CI for some project? They allow for purging the cache for a project -- https://github.com/travis-ci/travis.rb#cache.

I don't really care about this message if is benign, but is not, is a clear indication that the bug was triggered.

The message isn't the problem here. The problem is that the lock file hasn't been removed. That should be cleaned up.

---

My current workaround was to add sudo pip install --upgrade-strategy only-if-needed 'httplib2>=0.10.3' to configuration management scripts, with a note pointing to this issue. Still I will try to assure that this is fixed upstream as I don't fancy those workarounds.... we all know how nice is to drawn in them ;)

I think I don't get the context to understand this. ._.

Also, I imagine you know that sudo pip is a really bad idea?

[ssbarnea]

@pradyunsg How it comes that as soon I upgraded httplib2, I was able to successfully run the same command that was failing before?

Yep, installing as root from pip discouraged on RHEL systems, but is still used for special cases, especially for temporary workarounds until we get a new RPM ready at the end of the pipe.

[pradyunsg]

Is there still a lock file somewhere? Maybe that got removed?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.