parse_requirements fails with environment markers

Originally reported by: dims (Bitbucket: dims, GitHub: dims)

---

Just a simple call like the following fails:

#!bash

pkg_resources.parse_requirements('Twis-Ted>=1.2-1;python_version==\'2.7\'')

Found this when trying to run tox -e py27 on https://github.com/openstack/oslo.privsep in a mac environment with latest setuptools.

#!bash

tox -e py27
GLOB sdist-make: /Users/dims/openstack/oslo/oslo.privsep/setup.py
ERROR: invocation failed (exit code 1), logfile: /Users/dims/openstack/oslo/oslo.privsep/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/local/opt/python/bin/python2.7', local('/Users/dims/openstack/oslo/oslo.privsep/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/Users/dims/openstack/oslo/oslo.privsep/.tox/dist')]
env: None

error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in enum34;python_version=='2.7' or python_version=='2.6' at ;python_version=='2.7' or python_version=='2.6'

ERROR: FAIL could not package project - v = InvocationError('/usr/local/opt/python/bin/python2.7 /Users/dims/openstack/oslo/oslo.privsep/setup.py sdist --formats=zip --dist-dir /Users/dims/openstack/oslo/oslo.privsep/.tox/dist (see /Users/dims/openstack/oslo/oslo.privsep/.tox/log/tox-0.log)', 1)

If i edit the line, i can get past the error:
https://github.com/openstack/oslo.privsep/blob/master/requirements.txt#L10

However since OpenStack uses this syntax a lot, it would be great to add support for this syntax:
https://github.com/openstack/requirements/blob/master/global-requirements.txt

Here are some tracebacks to pin point the problem (line numbers may vary!)

#!bash

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/setuptools/dist.py", line 125, in check_requirements
    list(pkg_resources.parse_requirements(value))
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2930, in parse_requirements
    "version spec")
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2895, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/470

Original comment by tysonclugg (Bitbucket: tysonclugg, GitHub: tysonclugg):

---

Are you sure that this isn't because of the following virtualenv bugs, all of which refer to the fact that virtualenv bundles setuptools 0.18.2 and installs that rather than the latest available version (or at least the version you have installed)?

• Can't download setuptools virtualenv#807
• Make use of recent cached setuptools/pip/wheel wheels when creating virtualenv#801
• [Rewrite] Embedding pip and setuptools virtualenv#717
• Automatically download and cache new pip/setuptools versions virtualenv#781
• Allow to use non-bundled versions of other projects virtualenv#580
• Please add an option to install latest setuptools/pip instead of embedded ones virtualenv#563
• virtualenv "pinned" to setuptools-0.9.8 even when a superior setuptools is installed virtualenv#491

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

@tysonclugg, I don't think it's a bug with 18.2. I don't think setuptools supports environment markers except in 'extras'.

The way to specify a requirement like 'Twis-Ted>=1.2-1;python_version==\'2.7\'' in setuptools is to have an extras_require like so:

    extras_require={
        ':python_version="2.7"': ['Twis-Ted>=1.2-1'],
    },

So it could be said that the bug lies in pbr, which isn't properly translating a requirements.txt format back into the expected syntax that setuptools requires.

In any case, it's not a simple "fix", but rather a feature request, and a non-trivial one, because it would involve supporting a new syntax that's already supported through a different mechanism.

[bittner]

@dims @tysonclugg @jaraco:

Can't setuptools support requirement specifiers (environment markers) in install_requires? There should be no need for extras_require when you specify them in install_requires.

pip works, setuptools breaks

I have a requirements.txt file in every project, which is also read by the project's setup.py. Hence, my users / developers can choose to run python setup.py install (develop) or run pip install -r requirements.txt. Which ever comes more natural to them.

Unfortunately, using environment markers is supported by pip but not by setuptools:

Using pip (with Python 3):

$ pip install -r requirements.txt 
Ignoring dnspython: markers "python_version < '3'" don't match your environment
Installing collected packages: dnspython3
Successfully installed dnspython3-1.12.0

Using setuptools (with Python 3):

$ python setup.py develop
error in GuuruBackoffice setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in dnspython3 ;python_version >= '3' at  ;python_version >= '3'

Related code samples

# -- FILE: requirements.txt
dnspython3 ;python_version >= '3'
dnspython ;python_version < '3'

# -- FILE: setup.py
from os.path import abspath, dirname, join
from setuptools import setup, find_packages

def read_file(filename):
    with open(join(abspath(dirname(__file__)), filename)) as f:
        return f.read()

setup(
    # ...
    install_requires=read_file('requirements.txt'),
    # ...
)

[bittner]

Can't we implement the same (or a matching) requirements parsing as pip for markers and deprecate extras_require?

This way a just single feature/keyword does it all for requirements. And we do away with some confusion, because pip's and setuptools' behavior match more closely.

[jaraco]

Reading the history, in Setuptools 20.5 and more stable in 20.6.8, Setuptools should support environment markers in install_requires and others. I do suspect that this feature can't be widely adopted until older versions of Setuptools have worked their way through retirement. @bittner, can you confirm that the later releases work as you would expect?

Deprecate extras_require?

Perhaps, but I struggle to think of how this is done without first upgrading the world to Setuptools 20.6.8 or later.

[bittner]

@jaraco You're right, using the marker's in requirements.txt works for, e.g., version 21.0.0, which I have installed now, that's awesome!

I'd say you have to add deprecation code to setuptools one days, or you'll never be able to remove a (confusing) feature.

[AraHaan]

Wait there is an python_version arg. Wow this actually could be useful.