pipenv 2022.9.20 broken

[scastria]

When I run: pipenv install --system --skip-lock
it doesn't install anything.

My Pipfile is:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-XXXXXXXXXX.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
"boxsdk[jwt]" = "==3.4.0"
my_utils = {version="==0.0.70", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3"

If I use pipenv 2022.9.8 (the previous version), everything works fine.

[matteius]

@scastria We changed dependencies for the Pipfile to plette -- so far I am unable to reproduce this though -- environment variable expansion seems to work. I am wondering if you need to make aws an environment variable as well for the expansion to work. If you pass --versbose do you get anymore details on where the error occurs?

[scastria]

Unfortunately, there is no error. It succeeds but installs no dependencies at all. None from pypi and none from my private source. Adding verbose doesn't add much. I don't think it is an expansion issue because if that was the case, I should get an error stating it can't find my_utils. Here is the output:

placer(main) > pipenv install --skip-lock --verbose
Using python: 3
Path to python: /opt/homebrew/bin/python3
Creating a virtualenv for this project...
Pipfile: /Users/shawncastrianni/GIT/placer/Pipfile
Using /opt/homebrew/bin/python3 (3.10.6) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.10.6.final.0-64 in 108ms
  creator CPython3Posix(dest=/Users/shawncastrianni/.local/share/virtualenvs/placer-eL1JtOAv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/shawncastrianni/Library/Application Support/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/shawncastrianni/.local/share/virtualenvs/placer-eL1JtOAv
Installing dependencies from Pipfile...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
placer(main) > pipenv run pip list
Package    Version
---------- -------
pip        22.2.2
setuptools 65.2.0
wheel      0.37.1

[scastria]

Hopefully, you have access to docker. That should be a good way to allow you to reproduce since you will start with exactly the same starting point as me. Here is the Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"

[dev-packages]

[requires]
python_version = "3"

Here is the Dockerfile:

FROM public.ecr.aws/bitnami/python:3.7
WORKDIR /app
COPY Pipfile .
RUN pip install pipenv
RUN pipenv install --system --skip-lock
ENTRYPOINT ["pip", "list"]

If you run the docker container, it spits out the list of dependencies. You will see it does NOT list requests as a dependency since pipenv didn't actually do anything:

placer(main) > docker run --rm pipenv:2
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Package            Version
------------------ ---------
certifi            2022.9.14
distlib            0.3.6
filelock           3.8.0
importlib-metadata 4.12.0
pip                21.3.1
pipenv             2022.9.20
platformdirs       2.5.2
setuptools         63.4.3
typing_extensions  4.3.0
virtualenv         20.16.5
virtualenv-clone   0.5.7
zipp               3.8.1

[scastria]

If you then change the Dockerfile to install 2022.9.8 version of pipenv, like this:

FROM public.ecr.aws/bitnami/python:3.7
WORKDIR /app
COPY Pipfile .
RUN pip install pipenv==2022.9.8
RUN pipenv install --system --skip-lock
ENTRYPOINT ["pip", "list"]

and then run that new container, it works and shows requests in the list of dependencies:

placer(main) > docker run --rm pipenv:3
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Package            Version
------------------ ---------
certifi            2022.9.14
charset-normalizer 2.1.1
distlib            0.3.6
filelock           3.8.0
idna               3.4
importlib-metadata 4.12.0
pip                21.3.1
pipenv             2022.9.8
platformdirs       2.5.2
requests           2.28.1
setuptools         63.4.3
typing_extensions  4.3.0
urllib3            1.26.12
virtualenv         20.16.5
virtualenv-clone   0.5.7
zipp               3.8.1

[matteius]

@scastria That first example worked for me -- meaning it actually worked and did not reproduce the issue:
EDIT: Actually I didn't realize it was so subtle that requests is missing -- will dig deeper.

matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5367$ docker build .
Sending build context to Docker daemon  3.072kB
Step 1/6 : FROM public.ecr.aws/bitnami/python:3.7
3.7: Pulling from bitnami/python
1d8866550bdd: Pull complete 
4114eed3ce0d: Pull complete 
Digest: sha256:a7b88ba7d9759bb1d5cf96b65bf65e6f73a3f716ed730fd972eb15c6a10f781f
Status: Downloaded newer image for public.ecr.aws/bitnami/python:3.7
 ---> 82365911445d
Step 2/6 : WORKDIR /app
 ---> Running in 314403e6cd88
Removing intermediate container 314403e6cd88
 ---> 7dba8c25f337
Step 3/6 : COPY Pipfile .
 ---> 91b7332646cc
Step 4/6 : RUN pip install pipenv
 ---> Running in 5a33e6daf809
Collecting pipenv
  Downloading pipenv-2022.9.20-py2.py3-none-any.whl (3.3 MB)
Collecting virtualenv-clone>=0.2.5
  Downloading virtualenv_clone-0.5.7-py3-none-any.whl (6.6 kB)
Requirement already satisfied: setuptools>=36.2.1 in /opt/bitnami/python/lib/python3.7/site-packages/setuptools-63.4.3-py3.7.egg (from pipenv) (63.4.3)
Requirement already satisfied: virtualenv in /opt/bitnami/python/lib/python3.7/site-packages (from pipenv) (20.16.5)
Collecting certifi
  Downloading certifi-2022.9.14-py3-none-any.whl (162 kB)
Requirement already satisfied: platformdirs<3,>=2.4 in /opt/bitnami/python/lib/python3.7/site-packages (from virtualenv->pipenv) (2.5.2)
Requirement already satisfied: filelock<4,>=3.4.1 in /opt/bitnami/python/lib/python3.7/site-packages (from virtualenv->pipenv) (3.8.0)
Requirement already satisfied: importlib-metadata>=4.8.3 in /opt/bitnami/python/lib/python3.7/site-packages (from virtualenv->pipenv) (4.12.0)
Requirement already satisfied: distlib<1,>=0.3.5 in /opt/bitnami/python/lib/python3.7/site-packages (from virtualenv->pipenv) (0.3.6)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/bitnami/python/lib/python3.7/site-packages (from importlib-metadata>=4.8.3->virtualenv->pipenv) (4.3.0)
Requirement already satisfied: zipp>=0.5 in /opt/bitnami/python/lib/python3.7/site-packages (from importlib-metadata>=4.8.3->virtualenv->pipenv) (3.8.1)
Installing collected packages: virtualenv-clone, certifi, pipenv
Successfully installed certifi-2022.9.14 pipenv-2022.9.20 virtualenv-clone-0.5.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Removing intermediate container 5a33e6daf809
 ---> 23c1d441e525
Step 5/6 : RUN pipenv install --system --skip-lock
 ---> Running in d149460833da
Installing dependencies from Pipfile...
Removing intermediate container d149460833da
 ---> 1a96da89af5c
Step 6/6 : ENTRYPOINT ["pip", "list"]
 ---> Running in 241b9c67e6a9
Removing intermediate container 241b9c67e6a9
 ---> 53a40e19d800
Successfully built 53a40e19d800
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5367$ docker run 241b9c67e6a9
Unable to find image '241b9c67e6a9:latest' locally
docker: Error response from daemon: pull access denied for 241b9c67e6a9, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5367$ docker run 53a40e19d800
Package            Version
------------------ ---------
certifi            2022.9.14
distlib            0.3.6
filelock           3.8.0
importlib-metadata 4.12.0
pip                21.3.1
pipenv             2022.9.20
platformdirs       2.5.2
setuptools         63.4.3
typing_extensions  4.3.0
virtualenv         20.16.5
virtualenv-clone   0.5.7
zipp               3.8.1
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5367$ cat Dockerfile 
FROM public.ecr.aws/bitnami/python:3.7
WORKDIR /app
COPY Pipfile .
RUN pip install pipenv
RUN pipenv install --system --skip-lock
ENTRYPOINT ["pip", "list"]
matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-5367$ cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"

[dev-packages]

[requires]
python_version = "3"

[matteius]

This seems to be a regression in the --skip-lock flag.

[oz123]

I can confirm that :

$ pipenv install --skip-lock -v
Using python: 3
Path to python: /usr/bin/python3
Creating a virtualenv for this project...
Pipfile: /home/oznt/Software/pypa/pipenv/test-p/Pipfile
Using /usr/bin/python3 (3.10.6) to create virtualenv...
⠇ Creating virtual environment...created virtual environment CPython3.10.6.final.0-64 in 313ms
  creator CPython3Posix(dest=/home/oz123/.local/share/virtualenvs/test-p-gqlzztUu, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/oz123/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==65.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/oz123/.local/share/virtualenvs/test-p-gqlzztUu
Installing dependencies from Pipfile...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
$ pipenv run pip list
Package    Version
---------- -------
pip        22.2.2
setuptools 65.2.0
wheel      0.37.1

[matteius]

@scastria If you can, please try this branch, it seemed to fix it locally and all tests passed. We will have to release a new version of plette and vendor that in, but this branch contains those changes already for testing purposes.
#5368

[scastria]

Can you provide instructions on how to test with a branch of pipenv? I have not done that before.

[matteius]

Can you provide instructions on how to test with a branch of pipenv?

pip install git+https://github.com/pypa/pipenv.git@issue-5367b

[JNKielmann]

I had a similar issue running:
pipenv install --dev --skip-lock
I can confirm that installing the patched version you mentioned fixed it for me.

[matteius]

2022.9.21 has been released to address this issue.

[TurnrDev]

Hi, I hate to be the bearer of bad news, but it seems that 2022.10.9 reintroduced this bug, possibly with these PRs: sarugaku/plette#28 + #5366

[matteius]

@TurnrDev Could you provide updated reproduction steps?

[TurnrDev]

Absolutely, we had two git branches we were swtiching between, one had a requirement that the other didn't. Our devs then run pipenv install --dev --skip-lock --clean (as we don't commit the lockfile, we didn't see the point in generating it)

So, basically... I guess it could also be reproduced by directly editing the Pipfile to add a new depenency in the default section.

I tested on three versions. 2022.10.4, 2022.10.8 and 2023.3.20. It only worked in 2022.10.4 Specifically, the new requirement we added was inflection, but the bug was not isolated to that requirement. On 2022.10.8 and 2023.3.10, pipenv install --dev --skip-lock --clean would appear to complete as if no requirements were changed. On 2022.10.4, it would actually install the new requirement. Using --verbose mode revealed no extra infromation on 2022.10.8 and 2023.3.10

[TurnrDev]

I can run new tests and get the program outputs but I promise it's entirely useless. Just showing this regardless:

$ pipenv install --dev --skip-lock --clear --verbose
Loading .env environment variables...
Installing dependencies from Pipfile...
Installing dependencies from Pipfile...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

[matteius]

Shawn -- please have a look at #5653 and help me validate if this truly fixes the problem. Local testing seemed good, the CI is running the unit tests now, but I'd appreciate more eyes on it.

…

-Matt Davis

On Sun, Apr 16, 2023 at 7:22 PM Shawn Castrianni ***@***.***> wrote: Any update on this? I just tried the latest 2023.3.20 and it fails with pipenv install --skip-lock — Reply to this email directly, view it on GitHub <#5367 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADVFFBX5UMYFRNENLJGGFDXBR5KZANCNFSM6AAAAAAQRQJVKQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[scastria]

@matteius Sorry, I posted here by mistake as my problem is new.