New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

parse_requirements fails with environment markers #470

Closed
bb-migration opened this Issue Dec 4, 2015 · 7 comments

Comments

Projects
None yet
4 participants
@bb-migration

bb-migration commented Dec 4, 2015

Originally reported by: dims (Bitbucket: dims, GitHub: dims)


Just a simple call like the following fails:

#!bash

pkg_resources.parse_requirements('Twis-Ted>=1.2-1;python_version==\'2.7\'')

Found this when trying to run tox -e py27 on https://github.com/openstack/oslo.privsep in a mac environment with latest setuptools.

#!bash

tox -e py27
GLOB sdist-make: /Users/dims/openstack/oslo/oslo.privsep/setup.py
ERROR: invocation failed (exit code 1), logfile: /Users/dims/openstack/oslo/oslo.privsep/.tox/log/tox-0.log
ERROR: actionid: tox
msg: packaging
cmdargs: ['/usr/local/opt/python/bin/python2.7', local('/Users/dims/openstack/oslo/oslo.privsep/setup.py'), 'sdist', '--formats=zip', '--dist-dir', local('/Users/dims/openstack/oslo/oslo.privsep/.tox/dist')]
env: None

error in setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in enum34;python_version=='2.7' or python_version=='2.6' at ;python_version=='2.7' or python_version=='2.6'

ERROR: FAIL could not package project - v = InvocationError('/usr/local/opt/python/bin/python2.7 /Users/dims/openstack/oslo/oslo.privsep/setup.py sdist --formats=zip --dist-dir /Users/dims/openstack/oslo/oslo.privsep/.tox/dist (see /Users/dims/openstack/oslo/oslo.privsep/.tox/log/tox-0.log)', 1)

If i edit the line, i can get past the error:
https://github.com/openstack/oslo.privsep/blob/master/requirements.txt#L10

However since OpenStack uses this syntax a lot, it would be great to add support for this syntax:
https://github.com/openstack/requirements/blob/master/global-requirements.txt

Here are some tracebacks to pin point the problem (line numbers may vary!)

#!bash

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/setuptools/dist.py", line 125, in check_requirements
    list(pkg_resources.parse_requirements(value))
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2930, in parse_requirements
    "version spec")
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2895, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])

@bb-migration

This comment has been minimized.

bb-migration commented Dec 8, 2015

Original comment by tysonclugg (Bitbucket: tysonclugg, GitHub: tysonclugg):


Are you sure that this isn't because of the following virtualenv bugs, all of which refer to the fact that virtualenv bundles setuptools 0.18.2 and installs that rather than the latest available version (or at least the version you have installed)?

@bb-migration

This comment has been minimized.

bb-migration commented Dec 9, 2015

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):


@tysonclugg, I don't think it's a bug with 18.2. I don't think setuptools supports environment markers except in 'extras'.

The way to specify a requirement like 'Twis-Ted>=1.2-1;python_version==\'2.7\'' in setuptools is to have an extras_require like so:

    extras_require={
        ':python_version="2.7"': ['Twis-Ted>=1.2-1'],
    },

So it could be said that the bug lies in pbr, which isn't properly translating a requirements.txt format back into the expected syntax that setuptools requires.

In any case, it's not a simple "fix", but rather a feature request, and a non-trivial one, because it would involve supporting a new syntax that's already supported through a different mechanism.

@bittner

This comment has been minimized.

Contributor

bittner commented Apr 26, 2016

@dims @tysonclugg @jaraco:

Can't setuptools support requirement specifiers (environment markers) in install_requires? There should be no need for extras_require when you specify them in install_requires.

pip works, setuptools breaks

I have a requirements.txt file in every project, which is also read by the project's setup.py. Hence, my users / developers can choose to run python setup.py install (develop) or run pip install -r requirements.txt. Which ever comes more natural to them.

Unfortunately, using environment markers is supported by pip but not by setuptools:

Using pip (with Python 3):

$ pip install -r requirements.txt 
Ignoring dnspython: markers "python_version < '3'" don't match your environment
Installing collected packages: dnspython3
Successfully installed dnspython3-1.12.0

Using setuptools (with Python 3):

$ python setup.py develop
error in GuuruBackoffice setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in dnspython3 ;python_version >= '3' at  ;python_version >= '3'

Related code samples

# -- FILE: requirements.txt
dnspython3 ;python_version >= '3'
dnspython ;python_version < '3'
# -- FILE: setup.py
from os.path import abspath, dirname, join
from setuptools import setup, find_packages

def read_file(filename):
    with open(join(abspath(dirname(__file__)), filename)) as f:
        return f.read()

setup(
    # ...
    install_requires=read_file('requirements.txt'),
    # ...
)
@bittner

This comment has been minimized.

Contributor

bittner commented May 29, 2016

Can't we implement the same (or a matching) requirements parsing as pip for markers and deprecate extras_require?

This way a just single feature/keyword does it all for requirements. And we do away with some confusion, because pip's and setuptools' behavior match more closely.

@jaraco

This comment has been minimized.

Member

jaraco commented May 29, 2016

Reading the history, in Setuptools 20.5 and more stable in 20.6.8, Setuptools should support environment markers in install_requires and others. I do suspect that this feature can't be widely adopted until older versions of Setuptools have worked their way through retirement. @bittner, can you confirm that the later releases work as you would expect?

Deprecate extras_require?

Perhaps, but I struggle to think of how this is done without first upgrading the world to Setuptools 20.6.8 or later.

@jaraco jaraco closed this in ab46f6f May 29, 2016

@bittner

This comment has been minimized.

Contributor

bittner commented May 29, 2016

@jaraco You're right, using the marker's in requirements.txt works for, e.g., version 21.0.0, which I have installed now, that's awesome!

I'd say you have to add deprecation code to setuptools one days, or you'll never be able to remove a (confusing) feature.

@AraHaan

This comment has been minimized.

AraHaan commented Jan 1, 2017

Wait there is an python_version arg. Wow this actually could be useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment