Pipenv lock fails for packages with extras as of 2022.8.13

[clewingriffith]

Issue description

There was a regression in the ability to lock Pipfiles containing packages with extras on the 2022.8.13 release. This was working fine on versions prior to 2022.8.13. I believe this is as a result of the changes introduced in #5234

Expected result

Given the Pipfile in the replicate section, it should lock, showing

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (3e8e27)!

Actual result

[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 824, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 772, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1126, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 919, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 723, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Constraints cannot have extras

Steps to replicate

With the following Pipfile

[packages]
"uvicorn[standard]" = "~=0.18.2"

[dev-packages]
pytest = "~=7.1.1"

[requires]
python_version = "3.9"

Run pipenv lock

Note that there is a package with 'extras', namely uvicorn[standard]. This issue does not occur with normal packages.

The change to lock the packages first and then treat that lock as constraints for the dev packages seems to raise the exception that the extras are not allowed.

[matteius]

@dqkqd fyi -- do you have some cycles to help me think more about this at it relates to the constraint files work?

[dqkqd]

@matteius Yes, there's a bug here.

pipenv/pipenv/utils/dependencies.py

Lines 287 to 290 in 697f1a9

	def is_constraint(dep):
	# https://pip.pypa.io/en/stable/user_guide/#constraints-files
	# constraints must have a name, they cannot be editable, and they cannot specify extras.
	return dep.name and not dep.editable and not dep.extras

I will open a PR for this.

[dqkqd]

Function to check constraints from pip has InstallRequirement as parameter:

pipenv/pipenv/patched/pip/_internal/req/req_install.py

Lines 854 to 880 in 697f1a9

	def check_invalid_constraint_type(req: InstallRequirement) -> str:
	# Check for unsupported forms
	problem = ""
	if not req.name:
	problem = "Unnamed requirements are not allowed as constraints"
	elif req.editable:
	problem = "Editable requirements are not allowed as constraints"
	elif req.extras:
	problem = "Constraints cannot have extras"
	if problem:
	deprecated(
	reason=(
	"Constraints are only allowed to take the form of a package "
	"name and a version specifier. Other forms were originally "
	"permitted as an accident of the implementation, but were "
	"undocumented. The new implementation of the resolver no "
	"longer supports these forms."
	),
	replacement="replacing the constraint with a requirement",
	# No plan yet for when the new resolver becomes default
	gone_in=None,
	issue=8210,
	)
	return problem

While is_constraint use Requiement as parameter:

pipenv/pipenv/utils/dependencies.py

Lines 287 to 298 in 697f1a9

	def is_constraint(dep):
	# https://pip.pypa.io/en/stable/user_guide/#constraints-files
	# constraints must have a name, they cannot be editable, and they cannot specify extras.
	return dep.name and not dep.editable and not dep.extras
	constraints = []
	for dep_name, dep in deps.items():
	new_dep = Requirement.from_pipfile(dep_name, dep)
	if is_constraint(new_dep):
	c = new_dep.as_line().strip()
	constraints.append(c)
	return constraints

[matteius]

This is fixed in pipenv==2022.8.19

[clewingriffith]

Thankyou guys for the extremely quick turnaround on this.

[antoniomika]

In pipenv, version 2022.8.24, we're still seeing a deprecation notice with extras in the constraints:

Warning: /usr/local/lib/python3.7/site-packages/pipenv/patched/pip/_internal/req/req_install.py:879: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
  issue=8210,

And failures occurring if a file was used to install a package (which from what I can tell, is still supported). pip-compile seems to have a --strip-extras flag (something similar might be the proper solution?) but we still have the failing installs when using a file configuration setting.

[matteius]

@antoniomika Is the file nor marked editable = "true" in the Pipfile? We spotted a bug with #egg fragment getting appended to some file installs ... expecting a new release soon that fixes that. Do you have a stack trace?

@dqkqd Could you look into that PipDeprecationWarning when you get a chance?

[antoniomika]

Here's the stacktrace we were seeing:

[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 825, in _main
[ResolutionFailure]:       pre, clear, verbose, system, write, requirements_dir, packages, dev
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 780, in resolve_packages
[ResolutionFailure]:       requirements_dir=requirements_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 760, in resolve
[ResolutionFailure]:       req_dir=requirements_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 1098, in resolve_deps
[ResolutionFailure]:       req_dir=req_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 882, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 686, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Unnamed requirements are not allowed as constraints

We don't have editable set to true either, but I imagine that is also an issue with the new pip resolver. Here's the three that could cause a deprecation (or failure):

    if not req.name:
        problem = "Unnamed requirements are not allowed as constraints"
    elif req.editable:
        problem = "Editable requirements are not allowed as constraints"
    elif req.extras:
        problem = "Constraints cannot have extras"

from here. Let me know what else I can do to help!

[dqkqd]

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

pipenv/pipenv/utils/dependencies.py

Lines 281 to 283 in 0d86629

	problem = check_invalid_constraint_type(new_dep.as_ireq())
	if not problem:
	c = new_dep.as_line().strip()

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

pipenv/pipenv/patched/pip/_internal/req/req_install.py

Lines 854 to 880 in 0d86629

	def check_invalid_constraint_type(req: InstallRequirement) -> str:
	# Check for unsupported forms
	problem = ""
	if not req.name:
	problem = "Unnamed requirements are not allowed as constraints"
	elif req.editable:
	problem = "Editable requirements are not allowed as constraints"
	elif req.extras:
	problem = "Constraints cannot have extras"
	if problem:
	deprecated(
	reason=(
	"Constraints are only allowed to take the form of a package "
	"name and a version specifier. Other forms were originally "
	"permitted as an accident of the implementation, but were "
	"undocumented. The new implementation of the resolver no "
	"longer supports these forms."
	),
	replacement="replacing the constraint with a requirement",
	# No plan yet for when the new resolver becomes default
	gone_in=None,
	issue=8210,
	)
	return problem

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

[antoniomika]

pip-compile seems to just strip the extras from a constraint file, which should be okay.

Unnamed reqs (files) and editable ones might be a different story.

[matteius]

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

pipenv/pipenv/utils/dependencies.py

Lines 281 to 283 in 0d86629

	problem = check_invalid_constraint_type(new_dep.as_ireq())
	if not problem:
	c = new_dep.as_line().strip()

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

pipenv/pipenv/patched/pip/_internal/req/req_install.py

Lines 854 to 880 in 0d86629

	def check_invalid_constraint_type(req: InstallRequirement) -> str:
	# Check for unsupported forms
	problem = ""
	if not req.name:
	problem = "Unnamed requirements are not allowed as constraints"
	elif req.editable:
	problem = "Editable requirements are not allowed as constraints"
	elif req.extras:
	problem = "Constraints cannot have extras"
	if problem:
	deprecated(
	reason=(
	"Constraints are only allowed to take the form of a package "
	"name and a version specifier. Other forms were originally "
	"permitted as an accident of the implementation, but were "
	"undocumented. The new implementation of the resolver no "
	"longer supports these forms."
	),
	replacement="replacing the constraint with a requirement",
	# No plan yet for when the new resolver becomes default
	gone_in=None,
	issue=8210,
	)
	return problem

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

I don't think we need to rewrite the pip function, I think its actually warning us of valid errors. I think the issue is that we need to filter out certain requirements from the Pipfile if they are constraints, because it ends up confusing the resolver if they get included. Probably for the extras we can do something like:

new_dep = Requirement.from_pipfile(dep_name, dep).as_ireq()
if new_dep .extras:
    new_dep .extras = None
if not new_dep.name or new_dep.editble:
    continue
....

[dqkqd]

Because all of the packages from [packages] will be put into that function to filter out non-constraint packages. So if there is, for example editable package in [packages], the warning messages will be thrown out. But of course, those non-constraint packages are not used in constraint-file.