Use environment marker for conditional requirements

[Hao-Liu]

Managing different requirements files for different python version
requires much more efforts when the requirements became too complex.

This issue emerged when a required package stevedore decide not
supporting Python 2.6 since version 1.11.0.

>>> import stevedore
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/stevedore/__init__.py", line 23, in <module>
    LOG.addHandler(logging.NullHandler())
AttributeError: 'module' object has no attribute 'NullHandler

Pip provides a feature named Environment Markers
for requirements file to include specific packages conditionally which could
properly solve this issue.

This fix:

1. Use environment markers and merge python version specific
   requirements files;
2. Remove python version specific requirements files references in
   Makefile and Travis configuration;
3. Use pip install -r requirements.txt to replace per-line package
   installation;
4. Restrain package stevedore on Python 2.6 to 1.10.0.

Signed-off-by: Hao Liu hliu@redhat.com

[Hao-Liu]

@lmr, Thanks for pointing out this. Now fixed.

[ldoktor]

Hello @Hao-Liu we got to the same problem at roughly the same time. We have this in our TODO for the current sprint. Anyway there is yet one thing to do, we need to reorder the make requirements target as it first installs python26 and then the default requirements and it uses --upgrade, so older stevedore will be installed, but then upgraded.

I haven't looked at it yet, could you please test it and adjust the makefile so it works as expected? (I think reorder should do, but I'm not sure if --upgrade also downgrades and if it works well. It needs to be tested before merge)

[Hao-Liu]

Hi @ldoktor, You're right about this. But I don't think simply reorder the installation is very proper even if it could solve this issue. What if python2.7 need to upper-limit a dependence but python2.6 not? (I know it's just a theoretical situation)

I googled and found that there seems exists a way to merge python-version-related dependencies into one single requirements.txt via Environment Markers.

Thus we can merge requirements.txt and requirements-python26.txt into:

# Avocado functional requirements
# SSH lib (avocado.utils.remote)
fabric>=1.7.0
# multiplexer (avocado.core.tree)
PyYAML>=3.11
# vm plugin (avocado.core.plugins.vm)
libvirt-python>=1.2.9
# .tar.xz support (avocado.utils.archive)
pyliblzma>=0.5.3
# HTML report plugin (avocado.core.plugins.htmlresult)
pystache>=0.5.3
# REST client (avocado.core.restclient)
requests>=1.2.3
# stevedore for loading "new style" plugins
stevedore>=1.8.0; python_version >= '2.7'
stevedore>=1.8.0,<=1.10.0; python_version < '2.7'
argparse>=1.3.0; python_version < '2.7'
logutils>=0.3.3; python_version < '2.7'
importlib>=1.0.3; python_version < '2.7'
unittest2>=1.0.0; python_version < '2.7'

@ldoktor @lmr Do you think we should do it this way?

[ldoktor]

Nice, I didn't know about this. Does it work in python 2.6?

[ldoktor]

Hmm in examples they have python 2.4, so it probably does. Would you, @Hao-Liu, please take care of all requirements files?

[Hao-Liu]

Hi @ldoktor, I've merged python26 requirements files into the other one and changed .travis.yaml and Makefile accordingly. Could you plz help check this PR. Thanks.

[clebergnu]

@Hao-Liu this is really cool! Thanks a lot for coming up with this very useful, yet unknown to many, feature of requirements.txt.

[ldoktor]

Well, I guess it was too soon... I tried it on F21 and not even python-pip-1.5.6-3.fc21.noarch supports that. @Hao-Liu, did you tried that on RHEL? I guess it won't work...

[ldoktor]

Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 277, in run
    InstallRequirement.from_line(name, None))
  File "/usr/lib/python2.7/site-packages/pip/req.py", line 172, in from_line
    return cls(req, comes_from, url=url, prereleases=prereleases)
  File "/usr/lib/python2.7/site-packages/pip/req.py", line 70, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/usr/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2669, in parse
    reqs = list(parse_requirements(s))
  File "/usr/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2607, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "/usr/lib/python2.7/site-packages/pip/_vendor/pkg_resources.py", line 2585, in scan_list
    "Expected ',' or end-of-list in",line,"at",line[p:]
ValueError: ("Expected ',' or end-of-list in", "stevedore>=1.8.0,<=1.10.0; python_version < '2.7'", 'at', "; python_version < '2.7'")

Storing debug log for failure in /root/.pip/pip.log

When I look into vim /usr/lib/python2.7/site-packages/pip/req.py, on F23 pip 7.1.2 contains markers, but the F21 does not.

[Hao-Liu]

@ldoktor , I'm extremely sorry that this patch break fedora tests. But I tested this on RHEL6 & 7 actually. Since my python-pip package was get from EPEL and its version is python-pip-7.1.0-1.el7.noarch and python-pip-7.1.0-1.el6.noarch

[ldoktor]

I see, well requirements are intended for all-kinds of distros, with stock versions of pip. Please have a look at my PR. It's not as nice as this one, but should work with older pip.