test_venv: test_with_pip() failed on "AMD64 Fedora without threads 3.x" buildbot: urllib3 dependency requires the threading module

[vstinner]

BPO	19766
Nosy	@vsajip, @ncoghlan, @vstinner, @tiran, @dstufft
Files	urllib3_no_thread.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/dstufft'
closed_at = <Date 2013-12-21.15:21:19.460>
created_at = <Date 2013-11-25.09:41:00.194>
labels = ['type-bug', 'tests']
title = 'test_venv: test_with_pip() failed on "AMD64 Fedora without threads 3.x" buildbot: urllib3 dependency requires the threading module'
updated_at = <Date 2013-12-21.15:21:29.702>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2013-12-21.15:21:29.702>
actor = 'ncoghlan'
assignee = 'dstufft'
closed = True
closed_date = <Date 2013-12-21.15:21:19.460>
closer = 'ncoghlan'
components = ['Tests']
creation = <Date 2013-11-25.09:41:00.194>
creator = 'vstinner'
dependencies = []
files = ['32857']
hgrepos = []
issue_num = 19766
keywords = ['patch', 'buildbot']
message_count = 15.0
messages = ['204316', '204413', '204475', '204495', '205111', '205506', '205512', '205535', '205542', '205557', '205590', '205813', '206225', '206227', '206753']
nosy_count = 5.0
nosy_names = ['vinay.sajip', 'ncoghlan', 'vstinner', 'christian.heimes', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19766'
versions = ['Python 3.4']

[vstinner]

The line "from . import exceptions" of urllib3 failed:
https://github.com/shazow/urllib3/blob/master/urllib3/__init__.py#L22

It is strange because urllib3/exceptions.py is part of the urllib3 module.

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/5674/steps/test/logs/stdio

======================================================================
FAIL: test_with_pip (test.test_venv.EnsurePipTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/test_venv.py", line 299, in test_with_pip
    self.run_with_capture(venv.create, self.env_dir, with_pip=True)
subprocess.CalledProcessError: Command '['/tmp/tmpd8r2w88y/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/test/test_venv.py", line 305, in test_with_pip
    self.fail(msg)
AssertionError: Command '['/tmp/tmpd8r2w88y/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

**Subprocess Output**
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/ensurepip/__main__.py", line 66, in <module>
    main()
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/ensurepip/__main__.py", line 61, in main
    default_pip=args.default_pip,
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/ensurepip/__init__.py", line 92, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/home/buildbot/buildarea/3.x.krah-fedora/build/Lib/ensurepip/__init__.py", line 28, in _run_pip
    import pip
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/__init__.py", line 11, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/vcs/mercurial.py", line 9, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/download.py", line 22, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/requests/__init__.py", line 58, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/requests/utils.py", line 24, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/requests/compat.py", line 7, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/requests/packages/__init__.py", line 3, in <module>
  File "/tmp/tmpwzs5sx88/pip-1.5.rc1-py2.py3-none-any.whl/pip/_vendor/requests/packages/urllib3/__init__.py", line 22, in <module>
ImportError: cannot import name 'exceptions'

[ncoghlan]

Potentially another platform compatibility issue for pip's dependencies.

[vstinner]

In fact, it's an issue in the urllib3 library which require threads.

$ cd /home/haypo/pip/PIP/pip/_vendor/requests/packages
$ /home/haypo/prog/python/default/venv/bin/python -c 'import urllib3'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/haypo/pip/PIP/pip/_vendor/requests/packages/urllib3/__init__.py", line 24, in <module>
    from .poolmanager import PoolManager, ProxyManager, proxy_from_url
  File "/home/haypo/pip/PIP/pip/_vendor/requests/packages/urllib3/poolmanager.py", line 14, in <module>
    from ._collections import RecentlyUsedContainer
  File "/home/haypo/pip/PIP/pip/_vendor/requests/packages/urllib3/_collections.py", line 8, in <module>
    from threading import RLock
  File "/home/haypo/prog/python/default/Lib/threading.py", line 4, in <module>
    import _thread
ImportError: No module named '_thread'

Here is a patch for urllib3.

I reported the issue upstream:
urllib3/urllib3#289

[vstinner]

Cool, my patch was already applied upstream:

urllib3/urllib3@929f158

@nick: can you try to package a new wheel package for pip?

[dstufft]

The urllib3 in requests VCS was updated, I just need to bother Kenneth to make a new release of requests or update pip to an unreleased requests.

[ncoghlan]

bpo-19924 suggests we may need a new distlib as well

[dstufft]

Requests was released and pip updated it, I can release a new pip but it appears that perhaps distlib needs fixed before the without threads case is taken care of?

[vsajip]

I will look at doing a distlib update shortly - but there's another issue (bpo-19913) that might also require an update - it' not clear yet.

[vsajip]

I've updated distlib to use dummy_threading where threading isn't available - see

https://bitbucket.org/pypa/distlib/commits/029fee573900765729402203e39b2171d7ae0784

Can someone please test with this version vendored into pip to check that the failures no longer occur in a no-thread environment, before I do a distlib release?

[tiran]

Vinay, thanks for your fast response! :) bpo-19913 should be resolved, too. A couple of months ago several people complained about a new file that looked like a ZIP bomb. This virus warnings looks even more severe although it's probably a false positive. Could you try a new set of binaries without UPX? https://www.virustotal.com/ lets you scan the files with more than 40 programs at once and for free.

[vsajip]

This commit in distlib uses uncompressed launcher executables which pass the virustotal.com checks:

https://bitbucket.org/pypa/distlib/commits/e23c9e4fd3125fa88063de4dec80367b1ac82aff

[dstufft]

Vinay, I've verified that the current default branch of distlib works without threading when vendored in pip and these tests pass.

[vsajip]

I've released distlib 0.1.5 on PyPI. This release uses dummy_threading when threading isn't available.

[ncoghlan]

Not quite fixed yet - Donald, I think all the vendored dependencies have been updated now, would it be possible to spin and incorporate a pip 1.5rc2 somewhat soonish so we know what's still left to be addressed on the ensurepip side?

[ncoghlan]

Donald updated CPython to pip 1.5rc2, so test_venv is now passing without threading support:

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/5874/steps/test/logs/stdio