Cannot install a wheel built from an extra_requires with environment markers

[pombredanne]

• Pip version: 8.12
• Python version: 2.7.6
• Operating System: Ubuntu x64 14.04 desktop

Description:

A plain wheel built with python setup.py bdist_wheel and https://github.com/nexB/scancode-toolkit/blob/develop/setup.py#L111 cannot be installed by pip.

The built wheel is there https://github.com/pombredanne/conan/blob/master/thirdparty/prod/scancode_toolkit-2.0.0.dev1-py2-none-any.whl
I am using all the latest venv, pip, wheel and setuptools FWIW
note: this is all about extra_requires and environment markers and these extra_requires are to work around some cross os build issue on the latest lxml.
Some possibly related bugs:
#3893 #3829

What I've run:

And I get this error when I pip install the wheel

$ pip install --upgrade -vv thirdparty/prod/scancode_toolkit-2.0.0.dev1-py2-none-any.whl 
Processing ./thirdparty/prod/scancode_toolkit-2.0.0.dev1-py2-none-any.whl
Cleaning up...
Invalid requirement: 'lxml==3.6.4; linux in "sys_platform"'
Traceback (most recent call last):
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/req/req_install.py", line 78, in __init__
    req = Requirement(req)
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "'; linux '"

Exception information:
Traceback (most recent call last):
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/commands/install.py", line 310, in run
    wb.build(autobuilding=True)
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/wheel.py", line 750, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/req/req_set.py", line 370, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/req/req_set.py", line 669, in _prepare_file
    add_req(subreq)
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/req/req_set.py", line 639, in add_req
    wheel_cache=self._wheel_cache,
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/req/req_install.py", line 87, in __init__
    "Invalid requirement: '%s'\n%s" % (req, add_msg))
InstallationError: Invalid requirement: 'lxml==3.6.4; linux in "sys_platform"'
Traceback (most recent call last):
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/req/req_install.py", line 78, in __init__
    req = Requirement(req)
  File "/home/pombreda/w421/conan/local/lib/python2.7/site-packages/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "'; linux '"

[dholth]

That's very weird. I tried switching linux to double quotes, either way the code translates lxml==3.6.4; "linux" in sys_platform to lxml==3.6.4; linux in "sys_platform" (it has moved the quotes). In the wheel's METADATA they are fine, and pkg_resources seems to understand the requirements just fine.

[dstufft]

Does it happen on master?

[jamadden]

I have a wheel built with many environment markers in extras (https://github.com/zodb/relstorage/blob/master/setup.py#L117) and it seems to install OK:

$ python --version
Python 2.7.8 (10f1b29a2bd2, Feb 02 2015, 21:22:55)
[PyPy 2.5.0 with GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.56)]
$ pip --version
pip 8.1.2 from /.../site-packages (python 2.7)
$ pip install dist/RelStorage-2.0.0b7.dev0-py2.py3-none-any.whl
Processing ./dist/RelStorage-2.0.0b7.dev0-py2.py3-none-any.whl
...
Collecting ZEO<5.0,>=4.2.0; python_full_version < "2.7.9" (from RelStorage==2.0.0b7.dev0)
Collecting ZODB<5.0,>=4.4.2; python_full_version < "2.7.9" (from RelStorage==2.0.0b7.dev0)
...
Installing collected packages: ZODB, ZEO, RelStorage
Successfully installed RelStorage-2.0.0b7.dev0 ZEO-4.3.0 ZODB-4.4.3

I use a different quoting convention:

        ":python_full_version > '2.7.8'": [
            'ZODB >= 4.4.2',
            'ZEO >= 4.2.0',
        ],
        ":python_full_version < '2.7.9'": [
            # We must pin old versions prior to 2.7.9 because ZEO
            # 5 only runs on versions with good SSL support.
            'ZODB >= 4.4.2, <5.0',
            'ZEO >= 4.2.0, <5.0'
        ],

[pombredanne]

Sorry for the noise. Thanks to @agronholm for spotting the issue. It should have using "Linux" and not "linux" . I switched to using platform_system markers too in aboutcode-org/scancode-toolkit@e229862 and all is well .
Closing

[jaraco]

Yes, the issue happens on master.

$ git-id
d503c209a03b3399ca463031cc37f3d87eb32371
$ python -m pip install -t foo ~/m/setuptools/setuptools-782/dist/setuptools_bug_failing-1.0.0-py3-none-any.whl
Processing /Users/jaraco/Dropbox/code/main/setuptools/setuptools-782/dist/setuptools_bug_failing-1.0.0-py3-none-any.whl
Invalid requirement: 'lxml==3.6.4; darwin in "sys_platform"'
Traceback (most recent call last):
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/packaging/requirements.py", line 92, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/pyparsing.py", line 1172, in parseString
    raise exc
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/pyparsing.py", line 1162, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/pyparsing.py", line 1028, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/pyparsing.py", line 2462, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/pyparsing.py", line 1032, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/pyparsing.py", line 2265, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pip._vendor.pyparsing.ParseException: Expected stringEnd (at char 11), (line:1, col:12)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jaraco/Dropbox/code/public/pip/pip/req/req_install.py", line 78, in __init__
    req = Requirement(req)
  File "/Users/jaraco/Dropbox/code/public/pip/pip/_vendor/packaging/requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
pip._vendor.packaging.requirements.InvalidRequirement: Invalid requirement, parse error at "'; darwin'"

That wheel was built from a checkout of https://github.com/pombredanne/setuptool-782 and the wheel content was validated in pypa/setuptools#782.

I suggest re-opening this ticket.

[pombredanne]

based on @jaraco investigation it makes sense to reopen.

[dstufft]

I think it's fixed in packaging, but I need to cut a release of that and pull it into pip.

[pombredanne]

@dstufft if I can help with testing, please ping me here