Breaking changes from 2023.7.23 to 2023.8.20 when using pyproject.toml

[ddl-kfrench]

Issue description

We noticed CI jobs began to fail over the weekend when upstream pipenv was picked up to perform pipenv install --dev --deploy on a repository with no changes other than an upgrade from pipenv - 2023.7.23 to pipenv - 2023.8.20

After some debugging, we found it appears to be due to a breaking change in how dependencies are resolved in pyproject.toml between these versions of pipenv. Based on the diff (v2023.7.23...v2023.8.20#diff-d5fa37485876aa13851c69343a10f66e9c5b8820257447a0a0b1645776d09172R42), there were a fair amount of modifications made in how requirements are parsed, including deprecation of requirementslib.

Expected result

pipenv install should continue to work for dependencies whose versions are specified in pyproject.toml rather than just Pipfile

Actual result

redacted logs from successful step using pipenv version 2023.7.23

pipenv clean
pipenv install --deploy --dev

Loading .env environment variables...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
Success!

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
Success!

Updated Pipfile.lock (xxx)!
Loading .env environment variables...
Installing dependencies from Pipfile.lock (xx)...
Installing dependencies from Pipfile.lock (xx)...

redacted logs from failing step using pipenv version 2023.8.20

only including relevant sections specific to the private dependency where failures occur (stubbed as github.com/org/repo.git v1.0.0):

pipenv clean
pipenv install --deploy --dev

Loading .env environment variables...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Success!
Warning: INFO:pipenv.patched.pip._internal.operations.prepare:Obtaining file:///home/circleci/project (from -r /tmp/pipenv-wrujgjp8-requirements/pipenv-kyt__35u-constraints.txt (line 3))
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Checking if build backend supports build_editable: started
INFO:pipenv.patched.pip._internal.cli.spinners:Checking if build backend supports build_editable: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: started
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): started
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): finished with status 'done'
...
...
INFO:pipenv.patched.pip._internal.operations.prepare:Collecting common@ git+ssh://git@github.com/org/repo.git@v1.0.0 (from application==0.0.0->-r /tmp/pipenv-wrujgjp8-requirements/pipenv-kyt__35u-constraints.txt (line 3))
INFO:pipenv.patched.pip._internal.vcs.git:Cloning ssh://****@github.com/org/repo.git (to revision v1.0.0) to /tmp/pip-temp-oesk1ice/common_50cbdabbce754467a4c199345000bc82
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git' /tmp/pip-temp-oesk1ice/common_50cbdabbce754467a4c199345000bc82
INFO:pip.subprocessor:Running command git checkout -q f832c163be4915ae9ebef4590f7d2141d5cd65c1
INFO:pipenv.patched.pip._internal.vcs.git:Resolved ssh://****@github.com/org/repo.git to commit f832c163be4915ae9ebef4590f7d2141d5cd65c1
...
...
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing build dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: started
INFO:pipenv.patched.pip._internal.cli.spinners:Getting requirements to build wheel: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: started
INFO:pipenv.patched.pip._internal.cli.spinners:Installing backend dependencies: finished with status 'done'
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): started
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (pyproject.toml): finished with status 'done'
...
...
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (setup.py): started
INFO:pipenv.patched.pip._internal.cli.spinners:Preparing metadata (setup.py): finished with status 'done'
...
...
INFO:pipenv.patched.pip._internal.vcs.git:Cloning ssh://****@github.com/org/repo.git (to revision v1.0.0) to /tmp/tmpkdh9kolc
INFO:pip.subprocessor:Running command git clone --filter=blob:none 'ssh://****@github.com/org/repo.git' /tmp/tmpkdh9kolc
INFO:pip.subprocessor:Cloning into '/tmp/tmpkdh9kolc'...
INFO:pip.subprocessor:Running command git checkout -q f832c163be4915ae9ebef4590f7d2141d5cd65c1
INFO:pipenv.patched.pip._internal.vcs.git:Resolved ssh://****@github.com/org/repo.git to commit f832c163be4915ae9ebef4590f7d2141d5cd65c1
..
..
Updated Pipfile.lock (7e50a95cbc0abb105303e9a25682548342d710c984e31e9566e6aabcae59590d)!
Loading .env environment variables...
Installing dependencies from Pipfile.lock (59590d)...
Installing dependencies from Pipfile.lock (59590d)...
[pipenv.exceptions.InstallError]: Collecting common@ git+ssh://git@github.com/org/repo.git@v1.0.0@f832c163be4915ae9ebef4590f7d2141d5cd65c1 (from -r /tmp/pipenv-24ej7m9_-requirements/pipenv-i2yt9ua0-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Cloning ssh://****@github.com/org/platform-python-common.git@v1.0.0 (to revision f832c163be4915ae9ebef4590f7d2141d5cd65c1) to /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e
[pipenv.exceptions.InstallError]: Running command git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git@v1.0.0' /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e
[pipenv.exceptions.InstallError]:   fatal: remote error:
[pipenv.exceptions.InstallError]:    org/repo.git@v1.0.0 is not a valid repository name
[pipenv.exceptions.InstallError]:   Visit https://support.github.com/ for help
[pipenv.exceptions.InstallError]:   error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   × git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git@v1.0.0' /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e did not run successfully.
[pipenv.exceptions.InstallError]:   │ exit code: 128
[pipenv.exceptions.InstallError]:   ╰─> See above for output.
[pipenv.exceptions.InstallError]:   
[pipenv.exceptions.InstallError]:   note: This error originates from a subprocess, and is likely not a problem with pip.
[pipenv.exceptions.InstallError]: error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: × git clone --filter=blob:none --quiet 'ssh://****@github.com/org/repo.git@v1.0.0' /tmp/pip-install-br8go210/common_f90e1bae5bc44e1e91525d67ec63685e did not run successfully.
[pipenv.exceptions.InstallError]: │ exit code: 128
[pipenv.exceptions.InstallError]: ╰─> See above for output.
[pipenv.exceptions.InstallError]: 
[pipenv.exceptions.InstallError]: note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Couldn't install package: {}

the generated Pipfile.lock for common when running 2023.8.20 looks correct

        "common": {
            "git": "git+ssh://git@github.com/org/repo.git@v1.0.0",
            "ref": "f832c163be4915ae9ebef4590f7d2141d5cd65c1"
        },

we tested the latest (8.20) pipenv install on a different repository where pyproject.toml does not include dependency versions and they are only contained in Pipfile. in this scenario, pipenv install succeeds.

however, in the newly failing job where pyproject.toml does include versions, pipenv install --deploy fails.

example of pyproject.toml

the same pyproject.toml succeeds in 2023.7.23 and fails in 2023.8.20

[project]
name = "application"
version = "0.0.0"
dependencies = [
  "common @ git+ssh://git@github.com/org/repo.git@v1.0.0",
]

Steps to replicate

for a repository with a pyproject.toml file that has a git+ssh dependency, run pipenv install --deploy.

---

can't include pipenv --support -- relevant redacted outputs included above

[matteius]

@ddl-kfrench I cannot be certain yet, but it seems to be it may possibly also be fixed by #5845 -- if you have some bandwidth to check, that would be helpful.

[fraenkel]

@matteius It doesn't fix the issue because the problem occurs after parsing the pyproject and leaving the @ bit.

[matteius]

@fraenkel The thing that I don't understand is, pipenv is not actually responsible for parsing the pyproject.toml -- we are feeding the top level requirements into the pip resolver and the resolver is what is introspecting those other things.

and leaving the @ bit.

What does this mean exactly? There is an extra @ and there shouldn't be? Or that there isn't a @ and there should be ... or something else?

I am wondering too if the version of 0.0.0 could be related to the issue at all?

[matteius]

Actually this stands out as an issue:

the generated Pipfile.lock for common when running 2023.8.20 looks correct

        "common": {
            "git": "git+ssh://git@github.com/org/repo.git@v1.0.0",
            "ref": "f832c163be4915ae9ebef4590f7d2141d5cd65c1"
        },

The lock file has a ref which is the actual commit hash and the git has the ref as well -- I think this is not right. Did the new version of pipenv generate this lock file or was it from a prior version?

It seems when I generate a new lock file using a similar command: pipenv install git+ssh://git@github.com/pypa/pipenv.git@v2023.8.20

The new lock file entry does not have the ref in the vcs part:

        "pipenv": {
            "git": "git+ssh://git@github.com/pypa/pipenv.git",
            "markers": "python_version >= '3.7'",
            "ref": "96fe0e630fa873a26b8f31f172437f9259a0a457"
        },

This is another case where we do not have tests showing this old style lock entry, and nobody reported it during the testing window for that branch. I guess my position right now is regenerating the lock file should fix it, but if someone wants to take on trying to peel that specifier off the vcs part of the URL if it exists(on the install pip lines that get generated) in order to make it more backwards compatible, I am open to it, but I lack time resources to tack it on right now.

[fraenkel]

Even with your patch, I see the following

Writing supplied requirement line to temporary file: 'common@ git+ssh://git@github.com/cerebrotech/platform-python-common.git@v1.0.3@f832c163be4915ae9ebef4590f7d2141d5cd65c1'
Writing supplied requirement line to temporary file: 'domino-config@ git+ssh://git@github.com/cerebrotech/domino-config.git@v0.4.0@2f3733ac470c2709644cb73ec26777e06d280af1'

In my case, the generated Pipfile.lock still has the branch/tag on the git reference, like:
"git": "git+ssh://git@github.com/cerebrotech/domino-config.git@v0.4.0",

[matteius]

@fraenkel I believe then its because the Pipfile also has the @v0.4.0 style ref in the vcs URL and that is getting transferred over to the lock-file. We can try to add a patch for this case, but that is what I think is happening.

[matteius]

I just pushed another change to https://github.com/pypa/pipenv/pull/5845/files that may handle it being in the Pipfile (might still require a re-lock) -- probably a little early to report this since the test runner is just kicking off ...

[matteius]

Ok that prior commit looked good for the Pipfile part, I just pushed another change to that branch which should make it so you wouldn't have to re-lock.

[matteius]

Just cut release 2023.8.21 with the proposed fix.