Pipenv lock fails for packages with extras as of 2022.8.13 #5273

clewingriffith · 2022-08-18T16:19:49Z

Issue description

There was a regression in the ability to lock Pipfiles containing packages with extras on the 2022.8.13 release. This was working fine on versions prior to 2022.8.13. I believe this is as a result of the changes introduced in #5234

Expected result

Given the Pipfile in the replicate section, it should lock, showing

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (3e8e27)!

Actual result

[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 824, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 772, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 751, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1126, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 919, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "***.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 723, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Constraints cannot have extras

Steps to replicate

With the following Pipfile

[packages]
"uvicorn[standard]" = "~=0.18.2"

[dev-packages]
pytest = "~=7.1.1"

[requires]
python_version = "3.9"

Run pipenv lock

Note that there is a package with 'extras', namely uvicorn[standard]. This issue does not occur with normal packages.

The change to lock the packages first and then treat that lock as constraints for the dev packages seems to raise the exception that the extras are not allowed.

The text was updated successfully, but these errors were encountered:

matteius · 2022-08-18T16:31:39Z

@dqkqd fyi -- do you have some cycles to help me think more about this at it relates to the constraint files work?

dqkqd · 2022-08-18T20:42:45Z

@matteius Yes, there's a bug here.

pipenv/pipenv/utils/dependencies.py

Lines 287 to 290 in 697f1a9

    
           def is_constraint(dep): 
        
               # https://pip.pypa.io/en/stable/user_guide/#constraints-files 
        
               # constraints must have a name, they cannot be editable, and they cannot specify extras. 
        
               return dep.name and not dep.editable and not dep.extras

I will open a PR for this.

dqkqd · 2022-08-18T21:07:52Z

Function to check constraints from pip has InstallRequirement as parameter:

pipenv/pipenv/patched/pip/_internal/req/req_install.py

Lines 854 to 880 in 697f1a9

    
           def check_invalid_constraint_type(req: InstallRequirement) -> str: 
        
               # Check for unsupported forms 
        
               problem = "" 
        
               if not req.name: 
        
                   problem = "Unnamed requirements are not allowed as constraints" 
        
               elif req.editable: 
        
                   problem = "Editable requirements are not allowed as constraints" 
        
               elif req.extras: 
        
                   problem = "Constraints cannot have extras" 
        
               if problem: 
        
                   deprecated( 
        
                       reason=( 
        
                           "Constraints are only allowed to take the form of a package " 
        
                           "name and a version specifier. Other forms were originally " 
        
                           "permitted as an accident of the implementation, but were " 
        
                           "undocumented. The new implementation of the resolver no " 
        
                           "longer supports these forms." 
        
                       ), 
        
                       replacement="replacing the constraint with a requirement", 
        
                       # No plan yet for when the new resolver becomes default 
        
                       gone_in=None, 
        
                       issue=8210, 
        
                   ) 
        
               return problem

While is_constraint use Requiement as parameter:

pipenv/pipenv/utils/dependencies.py

Lines 287 to 298 in 697f1a9

    
           def is_constraint(dep): 
        
               # https://pip.pypa.io/en/stable/user_guide/#constraints-files 
        
               # constraints must have a name, they cannot be editable, and they cannot specify extras. 
        
               return dep.name and not dep.editable and not dep.extras 
        
           constraints = [] 
        
           for dep_name, dep in deps.items(): 
        
               new_dep = Requirement.from_pipfile(dep_name, dep) 
        
               if is_constraint(new_dep): 
        
                   c = new_dep.as_line().strip() 
        
                   constraints.append(c) 
        
           return constraints

matteius · 2022-08-19T12:25:54Z

This is fixed in pipenv==2022.8.19

clewingriffith · 2022-08-19T14:09:26Z

Thankyou guys for the extremely quick turnaround on this.

antoniomika · 2022-08-30T14:15:49Z

In pipenv, version 2022.8.24, we're still seeing a deprecation notice with extras in the constraints:

Warning: /usr/local/lib/python3.7/site-packages/pipenv/patched/pip/_internal/req/req_install.py:879: PipDeprecationWarning: DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement. Discussion can be found at https://github.com/pypa/pip/issues/8210
  issue=8210,

And failures occurring if a file was used to install a package (which from what I can tell, is still supported). pip-compile seems to have a --strip-extras flag (something similar might be the proper solution?) but we still have the failing installs when using a file configuration setting.

matteius · 2022-08-30T14:27:39Z

@antoniomika Is the file nor marked editable = "true" in the Pipfile? We spotted a bug with #egg fragment getting appended to some file installs ... expecting a new release soon that fixes that. Do you have a stack trace?

@dqkqd Could you look into that PipDeprecationWarning when you get a chance?

antoniomika · 2022-08-30T14:40:08Z

Here's the stacktrace we were seeing:

[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 825, in _main
[ResolutionFailure]:       pre, clear, verbose, system, write, requirements_dir, packages, dev
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 780, in resolve_packages
[ResolutionFailure]:       requirements_dir=requirements_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 760, in resolve
[ResolutionFailure]:       req_dir=requirements_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 1098, in resolve_deps
[ResolutionFailure]:       req_dir=req_dir,
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 882, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 686, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: Unnamed requirements are not allowed as constraints

We don't have editable set to true either, but I imagine that is also an issue with the new pip resolver. Here's the three that could cause a deprecation (or failure):

    if not req.name:
        problem = "Unnamed requirements are not allowed as constraints"
    elif req.editable:
        problem = "Editable requirements are not allowed as constraints"
    elif req.extras:
        problem = "Constraints cannot have extras"

from here. Let me know what else I can do to help!

dqkqd · 2022-08-30T14:43:23Z

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

pipenv/pipenv/utils/dependencies.py

Lines 281 to 283 in 0d86629

    
           problem = check_invalid_constraint_type(new_dep.as_ireq()) 
        
           if not problem: 
        
               c = new_dep.as_line().strip()

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

pipenv/pipenv/patched/pip/_internal/req/req_install.py

Lines 854 to 880 in 0d86629

    
           def check_invalid_constraint_type(req: InstallRequirement) -> str: 
        
               # Check for unsupported forms 
        
               problem = "" 
        
               if not req.name: 
        
                   problem = "Unnamed requirements are not allowed as constraints" 
        
               elif req.editable: 
        
                   problem = "Editable requirements are not allowed as constraints" 
        
               elif req.extras: 
        
                   problem = "Constraints cannot have extras" 
        
               if problem: 
        
                   deprecated( 
        
                       reason=( 
        
                           "Constraints are only allowed to take the form of a package " 
        
                           "name and a version specifier. Other forms were originally " 
        
                           "permitted as an accident of the implementation, but were " 
        
                           "undocumented. The new implementation of the resolver no " 
        
                           "longer supports these forms." 
        
                       ), 
        
                       replacement="replacing the constraint with a requirement", 
        
                       # No plan yet for when the new resolver becomes default 
        
                       gone_in=None, 
        
                       issue=8210, 
        
                   ) 
        
               return problem

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

antoniomika · 2022-08-30T14:45:30Z

pip-compile seems to just strip the extras from a constraint file, which should be okay.

Unnamed reqs (files) and editable ones might be a different story.

matteius · 2022-08-30T14:59:15Z

I see, function check_invalid_constraint_type is used to check if a package is valid as a constraint.

pipenv/pipenv/utils/dependencies.py

Lines 281 to 283 in 0d86629

problem = check_invalid_constraint_type(new_dep.as_ireq())

if not problem:

c = new_dep.as_line().strip()

That check_invalid_constraint_type throws out the warning even we are not using invalid package as constraint.

pipenv/pipenv/patched/pip/_internal/req/req_install.py

Lines 854 to 880 in 0d86629

def check_invalid_constraint_type(req: InstallRequirement) -> str:

# Check for unsupported forms

problem = ""

if not req.name:

problem = "Unnamed requirements are not allowed as constraints"

elif req.editable:

problem = "Editable requirements are not allowed as constraints"

elif req.extras:

problem = "Constraints cannot have extras"

if problem:

deprecated(

reason=(

"Constraints are only allowed to take the form of a package "

"name and a version specifier. Other forms were originally "

"permitted as an accident of the implementation, but were "

"undocumented. The new implementation of the resolver no "

"longer supports these forms."

),

replacement="replacing the constraint with a requirement",

# No plan yet for when the new resolver becomes default

gone_in=None,

issue=8210,

)

return problem

Rewrite that function is a solution but we have to keep track if pip changes it as well. Is there anyway we could use it without emitting any warning?

I don't think we need to rewrite the pip function, I think its actually warning us of valid errors. I think the issue is that we need to filter out certain requirements from the Pipfile if they are constraints, because it ends up confusing the resolver if they get included. Probably for the extras we can do something like:

new_dep = Requirement.from_pipfile(dep_name, dep).as_ireq()
if new_dep .extras:
    new_dep .extras = None
if not new_dep.name or new_dep.editble:
    continue
....

dqkqd · 2022-08-30T15:06:58Z

Because all of the packages from [packages] will be put into that function to filter out non-constraint packages. So if there is, for example editable package in [packages], the warning messages will be thrown out. But of course, those non-constraint packages are not used in constraint-file.

matteius · 2022-08-30T15:17:59Z

@dqkqd I see what you mean now, we are using that method to filter them out already, but thats causing the emitting of a deprecation warning because that method seems to expect that data was already cleaned -- probably we should just have our own method for this like you were suggesting, be it a similar copy or a simplified refactor.

That begs the question of what is causing resolution failures for @antoniomika -- can you elaborate on the file configuration setting you are using? Do you just mean local editable installs, or something else?

antoniomika · 2022-08-30T15:25:04Z

It was a package fork with some additions from a PR. Here's the line:

django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"}

matteius · 2022-08-30T15:38:33Z

@antoniomika It would appear you must have a conflicting requirement with something that specific zip requires, because I can install it with requests by themselves:

Have you tried running pipenv install --skip-lock and inspect pipenv graph?

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ pipenv install requests
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\pipenv-5273\Pipfile
Using default python from C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1520.0_x64__qbz5n2kfra8p0\python.exe (3.10.5) to create virtualenv...
[   =] Creating virtual environment...created virtual environment CPython3.10.5.final.0-64 in 2595ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\pipenv-5273-3M5UDgwN, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.3.0, wheel==0.37.1
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: \c\Users\matte\.virtualenvs\pipenv-5273-3M5UDgwN
Creating a Pipfile for this project...
Installing requests...
Adding requests to Pipfile's [packages]...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
           Building requirements...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (a290a1)!
Installing dependencies from Pipfile.lock (a290a1)...
  ================================ 0/0 - 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ vim Pipfile

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ pipenv install
Pipfile.lock (a290a1) out of date, updating to (10741c)...
Locking [packages] dependencies...
           Building requirements...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (10741c)!
Installing dependencies from Pipfile.lock (10741c)...
  ================================ 11/11 - 00:00:08
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"
django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"}

[dev-packages]

[requires]
python_version = "3.10"

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5273
$ cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "22b8f975edd17661ac607aa5a0741f6496d3b9e841c72b7a7411746eef10741c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "asgiref": {
            "hashes": [
                "sha256:1d2880b792ae8757289136f1db2b7b99100ce959b2aa57fd69dab783d05afac4",
                "sha256:4a29362a6acebe09bf1d6640db38c1dc3d9217c68e6f9f6204d72667fc19a424"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==3.5.2"
        },
        "async-timeout": {
            "hashes": [
                "sha256:2163e1640ddb52b7a8c80d0a67a08587e5d245cc9c553a74a847056bc2976b15",
                "sha256:8ca1e4fcf50d07413d66d1a5e416e42cfdf5851c981d679a09851a6853383b3c"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==4.0.2"
        },
        "certifi": {
            "hashes": [
                "sha256:84c85a9078b11105f04f3036a9482ae10e4621616db313fe045dd24743a0820d",
                "sha256:fe86415d55e84719d75f8b69414f6438ac3547d2078ab91b67e779ef69378412"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2022.6.15"
        },
        "charset-normalizer": {
            "hashes": [
                "sha256:5a3d016c7c547f69d6f81fb0db9449ce888b418b5b9952cc5e6e66843e9dd845",
                "sha256:83e9a75d1911279afd89352c68b45348559d1fc0506b054b346651b5e7fee29f"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==2.1.1"
        },
        "deprecated": {
            "hashes": [
                "sha256:43ac5335da90c31c24ba028af536a91d41d53f9e6901ddb021bcc572ce44e38d",
                "sha256:64756e3e14c8c5eea9795d93c524551432a0be75629f8f29e67ab8caf076c76d"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'",
            "version": "==1.2.13"
        },
        "django": {
            "hashes": [
                "sha256:031ccb717782f6af83a0063a1957686e87cb4581ea61b47b3e9addf60687989a",
                "sha256:032f8a6fc7cf05ccd1214e4a2e21dfcd6a23b9d575c6573cacc8c67828dbe642"
            ],
            "markers": "python_version >= '3.8'",
            "version": "==4.1"
        },
        "django-redis": {
            "file": "https://github.com/antoniomika/django-redis/archive/4.12.2.zip",
            "hashes": [
                "sha256:2b84c91a174f5cb06e26d9772567ae8752f6cfe090f2d3fff2c39fd77548183e"
            ],
            "version": "==4.12.1"
        },
        "idna": {
            "hashes": [
                "sha256:84d9dd047ffa80596e0f246e2eab0b391788b0503584e8945f2368256d2735ff",
                "sha256:9d643ff0a55b762d5cdb124b8eaa99c66322e2157b69160bc32796e824360e6d"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==3.3"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        },
        "redis": {
            "hashes": [
                "sha256:a52d5694c9eb4292770084fa8c863f79367ca19884b329ab574d5cb2036b3e54",
                "sha256:ddf27071df4adf3821c4f2ca59d67525c3a82e5f268bed97b813cb4fabf87880"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==4.3.4"
        },
        "requests": {
            "hashes": [
                "sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983",
                "sha256:8fefa2a1a1365bf5520aac41836fbee479da67864514bdb821f31ce07ce65349"
            ],
            "index": "pypi",
            "version": "==2.28.1"
        },
        "sqlparse": {
            "hashes": [
                "sha256:0c00730c74263a94e5a9919ade150dfc3b19c574389985446148402998287dae",
                "sha256:48719e356bb8b42991bdbb1e8b83223757b93789c00910a616a071910ca4a64d"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==0.4.2"
        },
        "tzdata": {
            "hashes": [
                "sha256:21f4f0d7241572efa7f7a4fdabb052e61b55dc48274e6842697ccdf5253e5451",
                "sha256:c3119520447d68ef3eb8187a55a4f44fa455f30eb1b4238fa5691ba094f2b05b"
            ],
            "markers": "sys_platform == 'win32'",
            "version": "==2022.2"
        },
        "urllib3": {
            "hashes": [
                "sha256:3fa96cf423e6987997fc326ae8df396db2a8b7c667747d47ddd8ecba91f4a74e",
                "sha256:b930dd878d5a8afb066a637fbb35144fe7901e3b209d1cd4f524bd0e9deee997"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5' and python_version < '4'",
            "version": "==1.26.12"
        },
        "wrapt": {
            "hashes": [
                "sha256:00b6d4ea20a906c0ca56d84f93065b398ab74b927a7a3dbd470f6fc503f95dc3",
                "sha256:01c205616a89d09827986bc4e859bcabd64f5a0662a7fe95e0d359424e0e071b",
                "sha256:02b41b633c6261feff8ddd8d11c711df6842aba629fdd3da10249a53211a72c4",
                "sha256:07f7a7d0f388028b2df1d916e94bbb40624c59b48ecc6cbc232546706fac74c2",
                "sha256:11871514607b15cfeb87c547a49bca19fde402f32e2b1c24a632506c0a756656",
                "sha256:1b376b3f4896e7930f1f772ac4b064ac12598d1c38d04907e696cc4d794b43d3",
                "sha256:21ac0156c4b089b330b7666db40feee30a5d52634cc4560e1905d6529a3897ff",
                "sha256:257fd78c513e0fb5cdbe058c27a0624c9884e735bbd131935fd49e9fe719d310",
                "sha256:2b39d38039a1fdad98c87279b48bc5dce2c0ca0d73483b12cb72aa9609278e8a",
                "sha256:2cf71233a0ed05ccdabe209c606fe0bac7379fdcf687f39b944420d2a09fdb57",
                "sha256:2fe803deacd09a233e4762a1adcea5db5d31e6be577a43352936179d14d90069",
                "sha256:3232822c7d98d23895ccc443bbdf57c7412c5a65996c30442ebe6ed3df335383",
                "sha256:34aa51c45f28ba7f12accd624225e2b1e5a3a45206aa191f6f9aac931d9d56fe",
                "sha256:36f582d0c6bc99d5f39cd3ac2a9062e57f3cf606ade29a0a0d6b323462f4dd87",
                "sha256:380a85cf89e0e69b7cfbe2ea9f765f004ff419f34194018a6827ac0e3edfed4d",
                "sha256:40e7bc81c9e2b2734ea4bc1aceb8a8f0ceaac7c5299bc5d69e37c44d9081d43b",
                "sha256:43ca3bbbe97af00f49efb06e352eae40434ca9d915906f77def219b88e85d907",
                "sha256:4fcc4649dc762cddacd193e6b55bc02edca674067f5f98166d7713b193932b7f",
                "sha256:5a0f54ce2c092aaf439813735584b9537cad479575a09892b8352fea5e988dc0",
                "sha256:5a9a0d155deafd9448baff28c08e150d9b24ff010e899311ddd63c45c2445e28",
                "sha256:5b02d65b9ccf0ef6c34cba6cf5bf2aab1bb2f49c6090bafeecc9cd81ad4ea1c1",
                "sha256:60db23fa423575eeb65ea430cee741acb7c26a1365d103f7b0f6ec412b893853",
                "sha256:642c2e7a804fcf18c222e1060df25fc210b9c58db7c91416fb055897fc27e8cc",
                "sha256:6a9a25751acb379b466ff6be78a315e2b439d4c94c1e99cb7266d40a537995d3",
                "sha256:6b1a564e6cb69922c7fe3a678b9f9a3c54e72b469875aa8018f18b4d1dd1adf3",
                "sha256:6d323e1554b3d22cfc03cd3243b5bb815a51f5249fdcbb86fda4bf62bab9e164",
                "sha256:6e743de5e9c3d1b7185870f480587b75b1cb604832e380d64f9504a0535912d1",
                "sha256:709fe01086a55cf79d20f741f39325018f4df051ef39fe921b1ebe780a66184c",
                "sha256:7b7c050ae976e286906dd3f26009e117eb000fb2cf3533398c5ad9ccc86867b1",
                "sha256:7d2872609603cb35ca513d7404a94d6d608fc13211563571117046c9d2bcc3d7",
                "sha256:7ef58fb89674095bfc57c4069e95d7a31cfdc0939e2a579882ac7d55aadfd2a1",
                "sha256:80bb5c256f1415f747011dc3604b59bc1f91c6e7150bd7db03b19170ee06b320",
                "sha256:81b19725065dcb43df02b37e03278c011a09e49757287dca60c5aecdd5a0b8ed",
                "sha256:833b58d5d0b7e5b9832869f039203389ac7cbf01765639c7309fd50ef619e0b1",
                "sha256:88bd7b6bd70a5b6803c1abf6bca012f7ed963e58c68d76ee20b9d751c74a3248",
                "sha256:8ad85f7f4e20964db4daadcab70b47ab05c7c1cf2a7c1e51087bfaa83831854c",
                "sha256:8c0ce1e99116d5ab21355d8ebe53d9460366704ea38ae4d9f6933188f327b456",
                "sha256:8d649d616e5c6a678b26d15ece345354f7c2286acd6db868e65fcc5ff7c24a77",
                "sha256:903500616422a40a98a5a3c4ff4ed9d0066f3b4c951fa286018ecdf0750194ef",
                "sha256:9736af4641846491aedb3c3f56b9bc5568d92b0692303b5a305301a95dfd38b1",
                "sha256:988635d122aaf2bdcef9e795435662bcd65b02f4f4c1ae37fbee7401c440b3a7",
                "sha256:9cca3c2cdadb362116235fdbd411735de4328c61425b0aa9f872fd76d02c4e86",
                "sha256:9e0fd32e0148dd5dea6af5fee42beb949098564cc23211a88d799e434255a1f4",
                "sha256:9f3e6f9e05148ff90002b884fbc2a86bd303ae847e472f44ecc06c2cd2fcdb2d",
                "sha256:a85d2b46be66a71bedde836d9e41859879cc54a2a04fad1191eb50c2066f6e9d",
                "sha256:a9a52172be0b5aae932bef82a79ec0a0ce87288c7d132946d645eba03f0ad8a8",
                "sha256:aa31fdcc33fef9eb2552cbcbfee7773d5a6792c137b359e82879c101e98584c5",
                "sha256:b014c23646a467558be7da3d6b9fa409b2c567d2110599b7cf9a0c5992b3b471",
                "sha256:b21bb4c09ffabfa0e85e3a6b623e19b80e7acd709b9f91452b8297ace2a8ab00",
                "sha256:b5901a312f4d14c59918c221323068fad0540e34324925c8475263841dbdfe68",
                "sha256:b9b7a708dd92306328117d8c4b62e2194d00c365f18eff11a9b53c6f923b01e3",
                "sha256:d1967f46ea8f2db647c786e78d8cc7e4313dbd1b0aca360592d8027b8508e24d",
                "sha256:d52a25136894c63de15a35bc0bdc5adb4b0e173b9c0d07a2be9d3ca64a332735",
                "sha256:d77c85fedff92cf788face9bfa3ebaa364448ebb1d765302e9af11bf449ca36d",
                "sha256:d79d7d5dc8a32b7093e81e97dad755127ff77bcc899e845f41bf71747af0c569",
                "sha256:dbcda74c67263139358f4d188ae5faae95c30929281bc6866d00573783c422b7",
                "sha256:ddaea91abf8b0d13443f6dac52e89051a5063c7d014710dcb4d4abb2ff811a59",
                "sha256:dee0ce50c6a2dd9056c20db781e9c1cfd33e77d2d569f5d1d9321c641bb903d5",
                "sha256:dee60e1de1898bde3b238f18340eec6148986da0455d8ba7848d50470a7a32fb",
                "sha256:e2f83e18fe2f4c9e7db597e988f72712c0c3676d337d8b101f6758107c42425b",
                "sha256:e3fb1677c720409d5f671e39bac6c9e0e422584e5f518bfd50aa4cbbea02433f",
                "sha256:ee2b1b1769f6707a8a445162ea16dddf74285c3964f605877a20e38545c3c462",
                "sha256:ee6acae74a2b91865910eef5e7de37dc6895ad96fa23603d1d27ea69df545015",
                "sha256:ef3f72c9666bba2bab70d2a8b79f2c6d2c1a42a7f7e2b0ec83bb2f9e383950af"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'",
            "version": "==1.14.1"
        }
    },
    "develop": {}
}

antoniomika · 2022-08-30T15:41:28Z

@matteius interesting. I went down that route and couldn't find anything specific. I also found pipenv==2022.8.5 works fine (prior to the constraint changes) which brought me here.

I'll keep digging, but at the least the deprecation fix would still be a nice to have.

matteius · 2022-08-30T15:45:15Z

@antoniomika are you locking dev dependencies?

antoniomika · 2022-08-30T15:46:44Z

We have both. I'm running a pipenv update.

matteius · 2022-08-30T15:52:44Z

@antoniomika There may be a package shared between default and dev that are on conflicting versions?

antoniomika · 2022-08-30T17:56:53Z

@matteius not that I know of, but I'm doing a deeper dive now. I have confirmed everything works fine with the older version but I'll update you once I know more.

matteius · 2022-08-30T21:37:39Z

@antoniomika if you see two package with the same name in the Pipfile.lock one in dev, and one in default, that might be a good indicator.

antoniomika · 2022-09-01T15:28:31Z

@matteius I was able to find the constraint that was the issue, there were two different versions. Is it possible to get that information outputted as an error?

I'm now seeing this issue with a different package:

{'req': None, 'comes_from': '-c /tmp/pipenv5itk0bj7requirements/pipenv-cbcnfu2m-constraints.txt (line 9)', 'constraint': True, 'editable': False, 'permit_editable_wheels': False, 'legacy_install_reason': None, 'source_dir': None, 'link': <Link https://github.com/antoniomika/channels_rabbitmq/archive/v1.3.2.zip>, 'original_link': <Link https://github.com/antoniomika/channels_rabbitmq/archive/v1.3.2.zip>, 'original_link_is_in_wheel_cache': False, 'download_info': None, 'local_file_path': None, 'extras': set(), 'markers': None, 'satisfied_by': None, 'should_reinstall': False, '_temp_build_dir': None, 'install_succeeded': None, 'install_options': [], 'global_options': [], 'hash_options': {}, 'config_settings': None, 'prepared': False, 'user_supplied': False, 'isolated': True, 'build_env': <pipenv.patched.pip._internal.build_env.NoOpBuildEnvironment object at 0x7f5f9728d7d0>, 'metadata_directory': None, 'pyproject_requires': None, 'requirements_to_check': [], 'pep517_backend': None, 'use_pep517': None, 'needs_more_preparation': False}
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 858, in <module>
    main()
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 853, in main
    dev=parsed.dev,
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 825, in _main
    pre, clear, verbose, system, write, requirements_dir, packages, dev
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 780, in resolve_packages
    requirements_dir=requirements_dir,
  File "/usr/local/lib/python3.7/site-packages/pipenv/resolver.py", line 760, in resolve
    req_dir=requirements_dir,
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 1102, in resolve_deps
    req_dir=req_dir,
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 886, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 685, in resolve
    self.constraints  # For some reason it is important to evaluate constraints before resolver context
  File "/usr/local/lib/python3.7/site-packages/pipenv/utils/resolver.py", line 650, in constraints
    self._constraints.sort(key=lambda ireq: ireq.name)
TypeError: '<' not supported between instances of 'NoneType' and 'str'

I outputted the actual constraint and it doesn't look like a name is associated with it (the output is above). Not sure if that's something that needs to be fixed or is another problem on our end (which I assume it is?).

matteius · 2022-09-01T15:33:34Z

That looks like a possible bug, somehow the Install requirement name is None. I'll think more about it. Cc @dqkqd

dqkqd · 2022-09-01T16:30:31Z

That looks like a possible bug, somehow the Install requirement name is None. I'll think more about it. Cc @dqkqd

Yes, it's from the commit d33f4ec.
I didn't aware that Install Requirement name could be None. To suppress this error, there's a dirty fix:

self._constraints.sort(key=lambda ireq: ireq.name or "")

At first, I thought package name is unique. Turn out I was wrong, when there are multiple no-name packages, they will be duplicated.
Now I'm not sure if just using name as sort criteria is stable in that case. Is there anything unique could be used to compare two Install Requirement?
If there isn't, something like the the path of requirements file: /tmp/random-dir/random-file.txt may enter as criteria, and because it's random, the sort might not be stable and issue like #5239, might happen again.

antoniomika · 2022-09-01T16:38:28Z

Does making the name the url not solve the issue as well (and removes the duplicate issue)? That wouldn't necessarily solve all cases though.

matteius · 2022-09-01T17:45:42Z

@dqkqd we had a report that the constraints issue was not solved even with your change: #4660 (comment)

I am going to open a PR to revert this since based on the prior report: https://github.com/pypa/pipenv/pull/5299/files

I also did some sorting in this PR yesterday but I think these are only pip constraints that get passed this way: https://github.com/pypa/pipenv/pull/5313/files#diff-3542045ed7f41738cd4519e02753c9e83af5654c56c124598f0fbbd82405ec49R134

matteius · 2022-09-01T18:03:40Z

Well reverting the commit @dqkqd made does cause the constraints to jump around a lot more when locking, probably we need to think through this more than a revert. I like the idea of making it safer, but defaulting to empty string seems problematic.

dqkqd · 2022-09-01T18:07:34Z

I knew the root of this issue now, sorting or not isn't the problem. It's the constraints with name is None is passed into resolver. Somehow this package django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"} passed the check is_constraint but have the name is None.

antoniomika · 2022-09-01T18:09:05Z

(for reference, the fork I had was unneeded and I removed that dependency in favor of the upstream version. The latest issue I had was with channels-rabbitmq = {file = "https://github.com/antoniomika/channels_rabbitmq/archive/v1.3.2.zip"})

matteius · 2022-09-01T18:12:49Z

@dqkqd Is it the is_constraints method do you think? If so I suspect we could also check if the InstallRequirement does not have a name in your is_constraints check. Also I see there is also a spot in requirementslib that does a weird is_constraint check: https://github.com/pypa/pipenv/blob/main/pipenv/vendor/requirementslib/models/dependencies.py#L241

dqkqd · 2022-09-01T18:39:40Z

I think it's the bug in requirementlib. Actually the result from new_dep.as_line() at line 284.

pipenv/pipenv/utils/dependencies.py

Lines 274 to 286 in f04071c

    
           def get_constraints_from_deps(deps): 
        
               """Get contraints from Pipfile-formatted dependency""" 
        
               def is_constraints(dep: InstallRequirement) -> bool: 
        
                   return dep.name and not dep.editable and not dep.extras 
        
               constraints = [] 
        
               for dep_name, dep in deps.items(): 
        
                   new_dep = Requirement.from_pipfile(dep_name, dep) 
        
                   if is_constraints(new_dep.as_ireq()): 
        
                       c = new_dep.as_line().strip() 
        
                       constraints.append(c) 
        
               return constraints

For example:
if I passed in the packages django-redis = {file = "https://github.com/antoniomika/django-redis/archive/4.12.2.zip"} the output I expected is django-redis @ https://github.com/antoniomika/django-redis/archive/4.12.2.zip as mentioned in
https://pip.pypa.io/en/stable/reference/requirements-file-format/#example

But the actual output is https://github.com/antoniomika/django-redis/archive/4.12.2.zip, and it doesn't contain name anymore, it's name is None now.

The name is disappeared because:
as_line call self.line_instance.get_line

pipenv/pipenv/vendor/requirementslib/models/requirements.py

Lines 2801 to 2805 in f04071c

    
           parts = self.line_instance.get_line( 
        
               with_prefix=True, 
        
               with_hashes=include_hashes, 
        
               with_markers=include_markers, 
        
               as_list=as_list,

self.line_instance.get_line check if package should have a name or not:

pipenv/pipenv/vendor/requirementslib/models/requirements.py

Lines 219 to 220 in f04071c

    
           if self.is_named: 
        
               line = self.name_and_specifier

And because it has url, so it's name is discarded from now.

pipenv/pipenv/vendor/requirementslib/models/requirements.py

Lines 753 to 760 in f04071c

    
           def is_named(self): 
        
               # type: () -> bool 
        
               return not ( 
        
                   self.is_file_url 
        
                   or self.is_url 
        
                   or self.is_file 
        
                   or self.is_vcs 
        
                   or self.is_direct_url

dqkqd · 2022-09-01T18:52:22Z

Or maybe I'm wrong, PipDeprecationWarning says Constraints are only allowed to take the form of a package name and a version specifier so it should not containts url. In that case we could easily fix that by using new_dep.is_named.

new_dep = Requirement.from_pipfile(dep_name, dep) 
    if new_dep.is_named and is_constraints(new_dep.as_ireq()): 
        c = new_dep.as_line().strip() 
        constraints.append(c)

But the documentation does not say that. So maybe it could have url as constraints? I tried using this constraints file with pip and it did work.

# constraints file
django-redis @ https://github.com/antoniomika/django-redis/archive/4.12.2.zip

matteius · 2022-09-02T08:24:03Z

@dqkqd I found this interesting and has a lot of context: pypa/pip#8210

That was a long read, but the conclusion is that "unnamed constraints are not allowed."

dqkqd mentioned this issue Aug 18, 2022

Issue 5273 unable to install extras because constraints are not recognized #5274

Merged

2 tasks

matteius closed this as completed in #5274 Aug 18, 2022

matteius reopened this Aug 30, 2022

dqkqd mentioned this issue Aug 30, 2022

Fixes checking constraint throws deprecation messages. #5309

Merged

2 tasks

oz123 closed this as completed in #5309 Aug 30, 2022

matteius reopened this Sep 1, 2022

matteius added the Type: Bug 🐛 This issue is a bug. label Sep 2, 2022

matteius mentioned this issue Sep 2, 2022

Apply fix to constraints generation based on fact that pip resolver does not allow unnamed constraints. #5321

Merged

oz123 closed this as completed in #5321 Sep 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pipenv lock fails for packages with extras as of 2022.8.13 #5273

Pipenv lock fails for packages with extras as of 2022.8.13 #5273

clewingriffith commented Aug 18, 2022

matteius commented Aug 18, 2022

dqkqd commented Aug 18, 2022

dqkqd commented Aug 18, 2022

matteius commented Aug 19, 2022

clewingriffith commented Aug 19, 2022

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Aug 30, 2022

dqkqd commented Aug 30, 2022 •

edited

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

dqkqd commented Aug 30, 2022

matteius commented Aug 30, 2022 •

edited

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022 •

edited

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Sep 1, 2022 •

edited

matteius commented Sep 1, 2022

dqkqd commented Sep 1, 2022 •

edited

antoniomika commented Sep 1, 2022

matteius commented Sep 1, 2022

matteius commented Sep 1, 2022

dqkqd commented Sep 1, 2022

antoniomika commented Sep 1, 2022

matteius commented Sep 1, 2022

dqkqd commented Sep 1, 2022 •

edited

dqkqd commented Sep 1, 2022 •

edited

matteius commented Sep 2, 2022 •

edited

Pipenv lock fails for packages with extras as of 2022.8.13 #5273

Pipenv lock fails for packages with extras as of 2022.8.13 #5273

Comments

clewingriffith commented Aug 18, 2022

Issue description

Expected result

Actual result

Steps to replicate

matteius commented Aug 18, 2022

dqkqd commented Aug 18, 2022

dqkqd commented Aug 18, 2022

matteius commented Aug 19, 2022

clewingriffith commented Aug 19, 2022

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Aug 30, 2022

dqkqd commented Aug 30, 2022 • edited

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

dqkqd commented Aug 30, 2022

matteius commented Aug 30, 2022 • edited

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022 • edited

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Aug 30, 2022

matteius commented Aug 30, 2022

antoniomika commented Sep 1, 2022 • edited

matteius commented Sep 1, 2022

dqkqd commented Sep 1, 2022 • edited

antoniomika commented Sep 1, 2022

matteius commented Sep 1, 2022

matteius commented Sep 1, 2022

dqkqd commented Sep 1, 2022

antoniomika commented Sep 1, 2022

matteius commented Sep 1, 2022

dqkqd commented Sep 1, 2022 • edited

dqkqd commented Sep 1, 2022 • edited

matteius commented Sep 2, 2022 • edited

dqkqd commented Aug 30, 2022 •

edited

matteius commented Aug 30, 2022 •

edited

matteius commented Aug 30, 2022 •

edited

antoniomika commented Sep 1, 2022 •

edited

dqkqd commented Sep 1, 2022 •

edited

dqkqd commented Sep 1, 2022 •

edited

dqkqd commented Sep 1, 2022 •

edited

matteius commented Sep 2, 2022 •

edited