pkg_resources.RequirementParseError with vcs requirement

[simon-weber]

With a requirements.txt with a vcs dependency like:

git+https://github.com/kennethreitz/requests@master#egg=requests==2.10.0

I get this traceback:

Traceback (most recent call last):
  File "/Users/simonweber/platform/venv-builder/bin/devpi-builder", line 11, in <module>
    sys.exit(main())
  File "/Users/simonweber/platform/venv-builder/lib/python2.7/site-packages/devpi_builder/cli.py", line 106, in main
    packages = requirements.read(args.requirements)
  File "/Users/simonweber/platform/venv-builder/lib/python2.7/site-packages/devpi_builder/requirements.py", line 41, in read
    _extract_project_version(requirement) for requirement in pkg_resources.parse_requirements(input_file)
  File "/Users/simonweber/platform/venv-builder/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2920, in parse_requirements
    "version spec")
  File "/Users/simonweber/platform/venv-builder/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2885, in scan_list
    raise RequirementParseError(msg, line, "at", line[p:])
pkg_resources.RequirementParseError: Expected version spec in git+https://github.com/kennethreitz/requests@master#egg=requests==2.10.0 at +https://github.com/kennethreitz/requests@master#egg=requests==2.10.0

But, pip can accept this format without trouble:

$ pip install -r requirements.txt
Collecting requests==2.10.0 from git+https://github.com/kennethreitz/requests@master#egg=requests==2.10.0 (from -r requirements.txt (line 1))
  Cloning https://github.com/kennethreitz/requests (to master) to /private/var/folders/4x/dfcw90sj1wg4q3q_r01tl81m0000gp/T/pip-build-aXfoLP/requests
Installing collected packages: requests
  Running setup.py install for requests
Successfully installed requests-2.10.0

Here's the output of pip freeze. It's from a fresh install of devpi-builder and devpi, so I don't expect it to be the culprit:

$ pip freeze
You are using pip version 7.0.1, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
apipkg==1.4
beautifulsoup4==4.4.1
bleach==1.4.3
Chameleon==2.24
check-manifest==0.31
defusedxml==0.4.1
devpi==2.2.0
devpi-builder==2.2.0
devpi-client==2.6.3
devpi-common==3.0.0
devpi-plumber==0.2.15
devpi-server==4.0.0
devpi-web==3.1.0
docutils==0.12
execnet==1.4.1
html5lib==0.9999999
itsdangerous==0.24
junit-xml==1.6
PasteDeploy==1.5.2
pkginfo==1.3.2
pluggy==0.3.1
py==1.4.31
Pygments==2.1.3
pyramid==1.7
pyramid-chameleon==0.3
readme-renderer==0.7.0
repoze.lru==0.6
requests==2.10.0
six==1.10.0
tox==2.3.1
translationstring==1.3
twitter.common.contextutil==0.3.5
twitter.common.dirutil==0.3.5
twitter.common.lang==0.3.5
venusian==1.0
virtualenv==15.0.2
waitress==0.9.0
WebOb==1.6.1
wheel==0.24.0
Whoosh==2.7.4
zope.deprecation==4.1.2
zope.interface==4.1.3

For this example, I can get around it by installing, freezing, and then using the resulting requirements.txt, but I don't expect that to work if the package isn't available on pypi.

[simon-weber]

For what it's worth, I guess a requirements file like this wouldn't even work with devpi to begin with, since it wouldn't try the index before cloning. The install/freeze workaround would work, I think, but could get hairy in cases where the version overlaps with a different one on pypi (since we're configured to fall back to pypi if our devpi is down).

[xhochy]

By using the version indicator in the #egg= section we should be able determine the version to match with (note that per https://pip.readthedocs.io/en/stable/reference/pip_install/#vcs-support the version is appended here with a - instead).

To still use the pkg_resources packages, we would need to pre-process the requirements file and remove all sections that match git\+[^#]#egg. Not sure how hacky that is.

In my case the line looks like git+https://github.com/scipy/scipy.git@faeb248c6b99e0751ce6169fc2af87aebd2981f9#egg=scipy-0.19.0.dev0+faeb248

[jayvdb]

You might like to try https://github.com/di/pip-api to solve this