pip install on latest ubuntu fails with error

[rotemseekingalpha]

Description:
github actions for python 3.8 fails with the following error:

ERROR: Invalid requirement: 'Warning: The lock flag' (from line 2 of requirements.txt)

Action version:

runs-on: ubuntu-latest (python-version: [ '3.8' ])
- name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python-version }}

Platform:

• Ubuntu)
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:

Repro steps:
Create a repo with Pipfile, add dependencies and lock the Pipfile.
run:

python -m pip install --upgrade pip setuptools pipenv virtualenv PyYAML flake8 pylint nose coverage
pipenv --python '3.8' lock -r > requirements.txt
pip install -r requirements.txt

Expected behavior:
pip should install all required packages

Actual behavior:
pip fails with thee following error:

ERROR: Invalid requirement: 'Warning: The lock flag' (from line 2 of requirements.txt)

[dmitry-shibanov]

Hello @rotemseekingalpha. Thank you for your report. I've tried to reproduce the issue, but I can't reproduce the same error message. For me it looks like the issue with invalid file context.

Could you please provide a link to the public repository to reproduce the issue ?

[jaredthecoder]

I see the same thing as of today (did not happen yesterday). My repository is private, but here is a sample of the Github workflow.

name: GitHub Actions
on: [create, push]
jobs:
  backend-ci:
    name: Backend CI
    runs-on: ubuntu-20.04
    env:
      ...
    services:
      ...
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v2
      - name: Set up Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.9
      - name: Setup PIP Cache
        uses: actions/cache@v2
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
     # THIS FAILS, all successes before here
      - name: Install Python Dependencies
        working-directory: backend
        run: |
          pip install pipenv
          pipenv lock --keep-outdated --requirements --dev > requirements.txt
          pip install -r requirements.txt

[jaredthecoder]

This looks related to this: pypa/pipenv#5091

The latest version of pipenv should not have this issue if installed with setup-python, but in this case, since both I and the other person with the issue here are installing pipenv ourselves, it may be because PyPi has not caught up yet. So I assume this can be closed and will just need pipenv to install.

[matteius]

Sorry for this regression -- there is work to convert everyone to the requirements command in pipenv and we didn't catch this going to stdout in the code review process. There is a PR that fixes this deprecation warning for the lock -r, and will be released sometime in May. I am wondering though, why generate a requirements.txt to install from which usually will not include hashes to validate against? If you have a Pipfile.lock, you can install from that using pipenv sync or pipenv install --deploy -- which I think would be recommended. I understand they are two separate issues, but I want to understand better the use case for CI generating a requirements file to install from rather than rely on the lock file. Thanks!

[singhpratyush]

A quick fix is sticking to the immediately previous version

python -m pip install --upgrade pip setuptools pipenv==2022.4.21 virtualenv PyYAML flake8 pylint nose coverage

---

Update:

You should consider using pipenv to install dependencies and not convert them to requirements.txt and install using pip (because this).

A good way of doing so can be pipenv install --system if you don't want to change how you run commands inside your environment.

[singhpratyush]

@matteius

why generate a requirements.txt to install

The reason I do it is because pypa/pipenv#356. The dependency installation needs to happen as a part of building a Docker image on the CI pipeline, and it times out when using pipenv to do it.

Here's a blog post I wrote quite a while back regarding it - https://medium.com/analytics-vidhya/docker-containers-for-pipenv-3be128f1444.

I haven't dug deep inside how cross-compatible this process is (as you mentioned requirements.txt does not include hashes to validate against), but it has worked fine for me for the last 2-3 years.

[matteius]

@singhpratyush

The reason I do it is because pypa/pipenv#356. The dependency installation needs to happen as a part of building a Docker image on the CI pipeline, and it times out when using pipenv to do it.

The issue you linked to is from 2019, I think a lot has changed since then. Furthermore you are doing a pipenv lock -r to generate your requirements which goes through the same locking and resolution process that used to have timeout issues, in order to generate a requirements file. If you already have the lock file you can speed this up by simply installing from it and skipping lock in your CI altogether, which would be preferable for security reasons. Simply run pipenv sync or pipenv install --deploy to install the dependencies from your lock file using pipenv and then you can uninstall pipenv just the same as you are in that article to reduce space.

I haven't dug deep inside how cross-compatible this process is (as you mentioned requirements.txt does not include hashes to validate against), but it has worked fine for me for the last 2-3 years.

It is less about compatibility and more about security. If you are generating your lock file on every CI run and installing requirements from that without hash checking, then you aren't verifying that the packages you install in the CI today match the exact versions that you installed yesterday. The version numbers will match, but the package contents could have changed either because it resolved to a different pypi server (DNS poisoning or otherwise), package confusion attack against a private package in the public pypi, or general bad actor published a different version of a package that was not pinned and that gets pulled in automatically. There are many ways this could happen and that is what the hash checking helps prevent against, but it doesn't provide protection if the CI regenerates the hashes every build, or discards them during the install phase.

[rotemseekingalpha]

@jaredthecoder, @singhpratyush, @matteius Thanks :)

[singhpratyush]

Thanks for taking the time and explaining @matteius, and debunking some of my misconceptions.

I modified the CI pipeline to completely use pipenv and things played out well. The time to run the build pipeline didn't change much. The command I had to use was pipenv install --system --deploy to make sure everything worked just like before and I didn't had to move to doing pipenv run ... with each command.

I will update the article with the latest information.

Cheers!

[dmitry-shibanov]

Hello everyone. Thank you for your help. Yesterday the new version of pipenv was released. Could you please confirm that everything works as expected ?

[singhpratyush]

@dmitry-shibanov It's working as expected on 2022.5.2.

Here's what now works that was failing before

RUN apt-get update && \
    apt-get -y install python3-dev ... && \
    pip3 install pipenv ... && \
    pipenv lock -r > requirements.txt && \
    pipenv lock --dev -r > dev-requirements.txt && \
    pip3 install -r requirements.txt && \
    ...

[dmitry-shibanov]

Hello everyone. For now I'm going to close the issue because it works as expected for 2022.5.2. If you have any concerns feel free to ping us.

[ianyoung]

I'm getting an error when using the following in my workflows yaml:

steps:
    - name: "Checkout repository"
      uses: actions/checkout@v3
    
    - name: "Setup Python"
      uses: actions/setup-python@v4
      with:
        python-version: '3.10' 
    - run: pip install pipenv
    - run: pipenv lock -r > requirements.txt

Is this still an issue for anyone else?

[matteius]

@ianyoung There was a deprecation warning around install -r for a number of releases, please use the newer pipenv requirements command.

[ianyoung]

Ah, I've not used it for a while. That explains things. Thanks for the quick reply @matteius.

For anyone else who needs it here's the updated instruction in the docs.