Specifying 'extras' with remote wheel causes exception

[elanou]

Issue description

Trying to install wheel from remote file with 'extras' specified in pipfile causes exception.

pipfile ex:
'spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}'

Expected result

Install wheel and extras

Actual result

Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 102, in init
req = REQUIREMENT.parseString(requirement_string)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pyparsing/core.py", line 1141, in parse_string
raise exc.with_traceback(None)
pipenv.patched.pip._vendor.pyparsing.exceptions.ParseException: Expected string_end, found '#' (at char 12), (line:1, col:13)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3101, in init
super(Requirement, self).init(requirement_string)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/packaging/requirements.py", line 104, in init
raise InvalidRequirement(
pipenv.patched.pip._vendor.packaging.requirements.InvalidRequirement: Parse error at "'#egg='": Expected string_end
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 275, in parse_line
req = Requirement.from_line(line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2684, in from_line
r = file_req_from_parsed_line(parsed_line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 3084, in file_req_from_parsed_line
"req": parsed_line.requirement,
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 475, in requirement
self.parse_requirement()
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1081, in parse_requirement
self._requirement = init_requirement(name) # type: PackagingRequirement
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/vendor/requirementslib/models/utils.py", line 192, in init_requirement
req = Requirement.parse(name)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3148, in parse
req, = parse_requirements(s)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3094, in parse_requirements
yield Requirement(line)
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/patched/pip/_vendor/pkg_resources/init.py", line 3103, in init
raise RequirementParseError(str(e))
pipenv.patched.pip._vendor.pkg_resources.RequirementParseError: Parse error at "'#egg='": Expected string_end
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 816, in
main()
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 802, in main
_main(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 782, in _main
resolve_packages(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 730, in resolve_packages
results, resolver = resolve(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 709, in resolve
return resolve_deps(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 1105, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 891, in actually_resolve_deps
resolver = Resolver.create(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 460, in create
constraints, skipped, index_lookup, markers_lookup = resolver.get_metadata(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 219, in get_metadata
req, req_idx, markers_idx = self.parse_line(
File "/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 287, in parse_line
raise ResolutionFailure(
pipenv.exceptions.ResolutionFailure: ERROR: Failed to resolve requirement from line: https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy-darwin#egg=[transformers]
✘ Locking Failed!

Steps to replicate

add below to pipfile:
'spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}'

---

Pipenv version: '2022.9.24'

Pipenv location: '/usr/local/Cellar/pipenv/2022.9.24/libexec/lib/python3.11/site-packages/pipenv'

Python location: '/usr/local/Cellar/pipenv/2022.9.24/libexec/bin/python3.11'

OS Name: 'posix'

User pip version: '22.2.2'

PEP 508 Information:

{'implementation_name': 'cpython',
'implementation_version': '3.11.0',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '22.1.0',
'platform_system': 'Darwin',
'platform_version': 'Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:30 PDT '
'2022; root:xnu-8792.41.9~2/RELEASE_ARM64_T8103',
'python_full_version': '3.11.0',
'python_version': '3.11',
'sys_platform': 'darwin'}

[matteius]

Can this be rechecked with pipenv==2022.12.17?

[oz123]

This still happens on the latest release.

 $ pipenv install -v
Pipfile.lock not found, creating...
Locking [packages] dependencies...
⠙ pipenv.exceptions.ResolutionFailure: ERROR: Failed to resolve requirement from line: 
$ pipenv --version
pipenv, version 2022.12.18.dev0
$ cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}

[pipenv]
allow_prereleases = true

Looks like a bug in requirementslib.

[oz123]

Adding some more information I found out trying to debug this:

 $ .../virtualenvs/foo-_rgbUsyh/bin/python /...pypa/pipenv/pipenv/resolver.py --pre --category packages --write /tmp/resolverskea3pd8.json --constraints-file /tmp/pipenvy9wg0apmconstraints.txt
> ...pipenv/pipenv/resolver.py(802)_main()
-> os.environ["PIP_PYTHON_PATH"] = str(sys.executable)
(Pdb) c
>.../pypa/pipenv/pipenv/utils/resolver.py(273)parse_line()
-> try:
(Pdb) c
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/resolver.py", line 738, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 1102, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 890, in actually_resolve_deps
[ResolutionFailure]:       resolver = Resolver.create(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 460, in create
[ResolutionFailure]:       constraints, skipped, index_lookup, markers_lookup = resolver.get_metadata(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 216, in get_metadata
[ResolutionFailure]:       req, req_idx, markers_idx = self.parse_line(
[ResolutionFailure]:   File "...pypa/pipenv/pipenv/utils/resolver.py", line 287, in parse_line
[ResolutionFailure]:       raise ResolutionFailure(
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Failed to resolve requirement from line: https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers]

[oz123]

OK. This is definitely a bug in requirementslib:

(Pdb) p line
"{'https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers]"
(Pdb) s
--Call--
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2669)from_line()
-> @classmethod
(Pdb) n
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2673)from_line()
-> if isinstance(line, InstallRequirement):
(Pdb) n
> /home/oznt/Software/pypa/pipenv/pipenv/vendor/requirementslib/models/requirements.py(2675)from_line()
-> parsed_line = Line(line)
(Pdb) l
2670 	    @lru_cache()
2671 	    def from_line(cls, line):
2672 	        # type: (AnyStr) -> Requirement
2673 	        if isinstance(line, InstallRequirement):
2674 	            line = format_requirement(line)
2675 ->	        parsed_line = Line(line)

[oz123]

Suspected cause of problem:

The constraints from:

spacy = {file = "https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl", extras = ["transformers"]}

are written to:

{'https://files.pythonhosted.org/packages/8a/2a/fe2870e216511ed12b9b5acf89228abc0c76832b9572501e0b895a9e129d/spacy-3.4.3-cp39-cp39-macosx_10_9_x86_64.whl#egg=spacy#egg=spacy[transformers] -i https://pypi.org/simple'}

[Blubberblub]

I stumbled over the same issue when trying to install a wheel for the ray library. From my pipfile:

ray = { extras = ['default'], file = "https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl" }

Gave me this error:

ERROR: Failed to resolve requirement from line: https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl#egg=ray#egg=ray[default]

Originally i tried to replicate this line for the installation through pip:

pip install "ray[default] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl"

When installing the url through the command line without extras the installations works:

pipenv install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp310-cp310-manylinux2014_x86_64.whl

pipenv, version 2022.12.19