Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Handle editable pip installs #2731

Merged
merged 12 commits into from Mar 13, 2023

Conversation

gabibguti
Copy link
Contributor

@gabibguti gabibguti commented Mar 8, 2023

What kind of change does this PR introduce?

Handle editable pip installs in Pinned-Dependencies check.

Description

When performing pip install -e <pkg>, the install can be considered secure if the package and it's dependencies are hash-pinned. The package can come from a local or remote source (VCS install). If the package is local, it is considered secure. If the package is remote, it needs to pinned by commit hash to be secure. We have no way to identify if the dependencies installed from requirements are hash-pinned. The flag --require-hashes does not work with -e flag. To guarantee we are not installing malicious dependencies, we can use --no-deps flag to not install the dependencies. That way the install can be considered secure.

Examples
secure pip install --no-deps -e git+https://github.com/username/repo.git@0123456789abcdef0123456789abcdef01234567#egg=package"
secure pip install --no-deps -e .
not secure pip install -e git+https://github.com/username/repo.git@0123456789abcdef0123456789abcdef01234567#egg=package"
not secure pip install --no-deps -e git+https://github.com/username/repo.git#egg=package"
References

(Is it a bug fix, feature, docs update, something else?)

What is the current behavior?

Currently, we don't handle editable pip installs. We suggest using --require-hashes as remediation but it's not possible.

What is the new behavior (if this is a feature change)?**

  • Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

Closes #2228

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

Accept pip installs with `-e` flag along with `--no-deps` flag in Pinned-Dependencies check. Editable pip installs will be accepted if coming from a local source or a remote Git source pinned by hash, as long as it's not installing dependencies.

@gabibguti gabibguti temporarily deployed to integration-test March 8, 2023 17:22 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Mar 8, 2023

Codecov Report

Merging #2731 (baae4da) into main (110e352) will increase coverage by 0.13%.
The diff coverage is 100.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2731      +/-   ##
==========================================
+ Coverage   49.78%   49.91%   +0.13%     
==========================================
  Files         156      156              
  Lines       11638    11669      +31     
==========================================
+ Hits         5794     5825      +31     
  Misses       5490     5490              
  Partials      354      354              

@gabibguti gabibguti temporarily deployed to integration-test March 8, 2023 17:37 — with GitHub Actions Inactive
@gabibguti gabibguti temporarily deployed to integration-test March 8, 2023 17:46 — with GitHub Actions Inactive
@gabibguti gabibguti marked this pull request as ready for review March 8, 2023 18:25
Copy link
Member

@di di left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice job!

Copy link
Contributor

@laurentsimon laurentsimon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

checks/raw/shell_download_validate.go Show resolved Hide resolved
@gabibguti gabibguti temporarily deployed to integration-test March 9, 2023 14:57 — with GitHub Actions Inactive
@gabibguti gabibguti temporarily deployed to integration-test March 10, 2023 22:05 — with GitHub Actions Inactive
@laurentsimon laurentsimon enabled auto-merge (squash) March 10, 2023 23:54
@laurentsimon
Copy link
Contributor

Please rebase and we'll be able to merge.

Editable pip installs (-e) should be considered secure if the package is installed from a local source or a remote source (VCS install) but pinned by commit hash. To keep the behaviour we have for normal pip installs, we need to guarantee the package dependencies are pinned by hash too. For normal pip installs, we verify that by using --require-hashes flag. Unfortunately, --require-hashes flag is not compatible with editable installs, so we use --no-deps flag to verify the dependencies are not installed since we can't verify if they are pinned.

Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
auto-merge was automatically disabled March 13, 2023 13:21

Head branch was pushed to by a user without write access

@gabibguti gabibguti temporarily deployed to integration-test March 13, 2023 13:22 — with GitHub Actions Inactive
@gabibguti
Copy link
Contributor Author

rebased

@laurentsimon laurentsimon enabled auto-merge (squash) March 13, 2023 19:24
@laurentsimon laurentsimon temporarily deployed to integration-test March 13, 2023 19:24 — with GitHub Actions Inactive
@laurentsimon laurentsimon merged commit 6ff94eb into ossf:main Mar 13, 2023
37 checks passed
@gabibguti gabibguti deleted the fix/pip-install-e-flag branch March 14, 2023 12:32
azeemshaikh38 added a commit to azeemshaikh38/scorecard that referenced this pull request Mar 14, 2023
commit 00da7a8be965c09d044f978d6b9eafee1350bd30
Author: Azeem Shaikh <azeemshaikh38@gmail.com>
Date:   Tue Mar 14 23:07:19 2023 +0000

    Pr comments

commit 1127dd9
Merge: 274448f 23bd295
Author: Azeem Shaikh <azeemshaikh38@gmail.com>
Date:   Wed Mar 15 04:23:32 2023 +0530

    Merge branch 'main' into go-git

commit 274448f
Author: Azeem Shaikh <azeemshaikh38@gmail.com>
Date:   Tue Mar 14 22:52:30 2023 +0000

    Initial implementation of go-git client

    Signed-off-by: Azeem Shaikh <azeemshaikh38@gmail.com>

commit 23bd295
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Mar 14 20:28:41 2023 +0000

    :seedling: Bump github/codeql-action from 2.2.4 to 2.2.6 (ossf#2741)

commit fc026ef
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Mar 14 17:04:31 2023 +0000

    :seedling: Bump github.com/google/ko from 0.12.0 to 0.13.0 in /tools (ossf#2742)

commit 2e04214
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Mar 14 14:02:34 2023 +0000

    :seedling: Bump tj-actions/changed-files from 35.6.2 to 35.7.0

    Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 35.6.2 to 35.7.0.
    - [Release notes](https://github.com/tj-actions/changed-files/releases)
    - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md)
    - [Commits](tj-actions/changed-files@5ce975c...bd376fb)

    ---
    updated-dependencies:
    - dependency-name: tj-actions/changed-files
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit e36b590
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Tue Mar 14 08:59:20 2023 -0500

    :seedling: Bump actions/cache from 3.3.0 to 3.3.1 (ossf#2740)

    Bumps [actions/cache](https://github.com/actions/cache) from 3.3.0 to 3.3.1.
    - [Release notes](https://github.com/actions/cache/releases)
    - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
    - [Commits](actions/cache@940f3d7...88522ab)

    ---
    updated-dependencies:
    - dependency-name: actions/cache
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 6ff94eb
Author: Gabriela Gutierrez <gabigutierrez@google.com>
Date:   Mon Mar 13 19:42:37 2023 +0000

    :bug: Handle editable pip installs (ossf#2731)

    * fix: Handle editable pip install

    Editable pip installs (-e) should be considered secure if the package is installed from a local source or a remote source (VCS install) but pinned by commit hash. To keep the behaviour we have for normal pip installs, we need to guarantee the package dependencies are pinned by hash too. For normal pip installs, we verify that by using --require-hashes flag. Unfortunately, --require-hashes flag is not compatible with editable installs, so we use --no-deps flag to verify the dependencies are not installed since we can't verify if they are pinned.

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * test: Editable pip install in GHA

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * test: Editable pip install in Dockerfile

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * test: Editable pip install in shell script

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * fix: Code complexity increase

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * fix: Simplify boolean return

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * docs: Add pip editable install references in comments

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * fix: Handle multiple packages in editable pip install

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * test: Multi editable pip install in GHA

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * test: Multi editable pip install in Dockerfile

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    * test: Multi editable pip install in shell script

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>

    ---------

    Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com>
    Co-authored-by: laurentsimon <64505099+laurentsimon@users.noreply.github.com>

commit 110e352
Author: raghavkaul <8695110+raghavkaul@users.noreply.github.com>
Date:   Mon Mar 13 11:13:50 2023 -0400

    ✨ Gitlab support: RepoClient (ossf#2655)

    * Add make targets and E2E test target for GitLab only

    Signed-off-by: Raghav Kaul <raghavkaul@google.com>

    * Add GitLab support to RepoClient

    Signed-off-by: Raghav Kaul <raghavkaul@google.com>

    * Build

    * Make target for e2e-gitlab-token
    * Only run Gitlab tests in CI that don't require a token

    Signed-off-by: Raghav Kaul <raghavkaul@google.com>

    * Add tests

    Signed-off-by: Raghav Kaul <raghavkaul@google.com>

    * Remove spurious printf

    Signed-off-by: Raghav Kaul <raghavkaul@google.com>

    * 🐛 Check OSS Fuzz build file for Fuzzing check (ossf#2719)

    * Check OSS-Fuzz using project list

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Use clients.RepoClient interface to perform the new OSS Fuzz check

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * wip: add eager client for better repeated lookup of projects

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Split lazy and eager behavior into different implementations.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Add tests and benchmarks

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Switch to always parsing JSON to determine if a project is present. The other approach of looking for a substring match would lead to false positives.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Add eager constructor to surface status file errors sooner.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Switch existing users to new OSS Fuzz client

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Mark old method as deprecated in the godoc

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * remove unused comment.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Use new OSS Fuzz client in e2e test.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * fix typo.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Fix potential path bug with test server.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Force include the two JSON files which were being ignored by .gitignore

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * trim the status json file

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    ---------

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    ---------

    Signed-off-by: Raghav Kaul <raghavkaul@google.com>
    Signed-off-by: Spencer Schrock <sschrock@google.com>
    Co-authored-by: Spencer Schrock <sschrock@google.com>

commit 5625dda
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat Mar 11 17:14:42 2023 +0000

    :seedling: Bump github.com/onsi/ginkgo/v2 from 2.8.3 to 2.9.0 in /tools

    Bumps [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) from 2.8.3 to 2.9.0.
    - [Release notes](https://github.com/onsi/ginkgo/releases)
    - [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md)
    - [Commits](onsi/ginkgo@v2.8.3...v2.9.0)

    ---
    updated-dependencies:
    - dependency-name: github.com/onsi/ginkgo/v2
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>

commit d591e38
Author: Spencer Schrock <sschrock@google.com>
Date:   Fri Mar 10 16:02:05 2023 -0800

    🌱  Add RepoClient re-use E2E tests. (ossf#2625)

    * Add e2e test for re-used repoclient.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Improve diff logging

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Skip scorecard e2e test during unit tests.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    * Fix linter.

    Signed-off-by: Spencer Schrock <sschrock@google.com>

    ---------

    Signed-off-by: Spencer Schrock <sschrock@google.com>

commit a7e81bb
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Mar 10 08:20:28 2023 -0600

    :seedling: Bump actions/cache from 3.2.6 to 3.3.0 (ossf#2738)

    Bumps [actions/cache](https://github.com/actions/cache) from 3.2.6 to 3.3.0.
    - [Release notes](https://github.com/actions/cache/releases)
    - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
    - [Commits](actions/cache@69d9d44...940f3d7)

    ---
    updated-dependencies:
    - dependency-name: actions/cache
      dependency-type: direct:production
      update-type: version-update:semver-minor
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: pip install -e . dinged for not using hashes, but it can't
3 participants