🌱 Add RepoClient re-use E2E tests. #2625

spencerschrock · 2023-01-30T18:43:03Z

What kind of change does this PR introduce?

e2e test
(Is it a bug fix, feature, docs update, something else?)

PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

No e2e tests which test for accumulation of results.

What is the new behavior (if this is a feature change)?**

Adds a simple test that the results for repository B are the same whether calculated:

B in isolation
A then B, reusing the repo client

Tests for the changes have been added (for bug fixes/features)

This e2e test was backported to 28b116f to test that it would have caught the bug.
It found issues with both Contributors and License accumulating results.

Contributors

  "Contributors" reason mismatch: "55 different organizations found -- score normalized to 10" vs "21 different organizations found -- score normalized to 10"
  "Contributors" reason mismatch: "21 different organizations found -- score normalized to 10" vs "55 different organizations found -- score normalized to 10"
  "Contributors" reason mismatch: "21 different organizations found -- score normalized to 10" vs "55 different organizations found -- score normalized to 10"
  "Contributors" reason mismatch: "55 different organizations found -- score normalized to 10" vs "21 different organizations found -- score normalized to 10"
  "Contributors" reason mismatch: "21 different organizations found -- score normalized to 10" vs "55 different organizations found -- score normalized to 10"
  "Contributors" reason mismatch: "55 different organizations found -- score normalized to 10" vs "21 different organizations found -- score normalized to 10"

License

"License" details mismatch:
      debug: 0-0
      info: 4-2
      warn: 0-0
"License" details mismatch:
      debug: 0-0
      info: 2-4
      warn: 0-0
"License" details mismatch:
      debug: 0-0
      info: 2-4
      warn: 0-0
"License" details mismatch:
      debug: 0-0
      info: 4-2
      warn: 0-0
"License" details mismatch:
      debug: 0-0
      info: 2-4
      warn: 0-0
"License" details mismatch:
      debug: 0-0
      info: 4-2
      warn: 0-0

Which issue(s) this PR fixes

Fixes #2533

Special notes for your reviewer

I'm not 100% certain why the results are being outputted multiple times.
The E2E test currently runs all scorecard checks 3 times, which results in the following increase in token API usage:

REST: 152 / 5000
Search: 6/30
graphql: 6/5000

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

NONE

spencerschrock

We could also structure this in a way such that it only runs on a certain make command, but not part of the normal E2E suite if we're concerned about the increase in API usage.

laurentsimon · 2023-01-30T20:16:08Z

We could also structure this in a way such that it only runs on a certain make command, but not part of the normal E2E suite if we're concerned about the increase in API usage.

how would the make command avoid increasing API usage?

spencerschrock · 2023-01-30T21:25:05Z

We could also structure this in a way such that it only runs on a certain make command, but not part of the normal E2E suite if we're concerned about the increase in API usage.

how would the make command avoid increasing API usage?

By making it available for contributors/maintainers to run on their own, without including it in the e2e suite that runs on every PR. Of course, the trade-off is remembering to run it when it is needed.

github-actions · 2023-02-10T02:40:21Z

Stale pull request message

Signed-off-by: Spencer Schrock <sschrock@google.com>

codecov · 2023-02-10T21:40:58Z

Codecov Report

Merging #2625 (76db929) into main (a7e81bb) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2625   +/-   ##
=======================================
  Coverage   49.86%   49.86%           
=======================================
  Files         156      156           
  Lines       11576    11576           
=======================================
  Hits         5772     5772           
  Misses       5454     5454           
  Partials      350      350

github-actions · 2023-02-21T02:02:35Z

Stale pull request message

naveensrinivasan · 2023-02-26T19:33:27Z

@spencerschrock A friendly ping.

spencerschrock · 2023-02-28T02:58:53Z

This test exposed a data race that has been present for a while, which will need to be fixed before these tests can be merged.

Signed-off-by: Spencer Schrock <sschrock@google.com>

commit 00da7a8be965c09d044f978d6b9eafee1350bd30 Author: Azeem Shaikh <azeemshaikh38@gmail.com> Date: Tue Mar 14 23:07:19 2023 +0000 Pr comments commit 1127dd9 Merge: 274448f 23bd295 Author: Azeem Shaikh <azeemshaikh38@gmail.com> Date: Wed Mar 15 04:23:32 2023 +0530 Merge branch 'main' into go-git commit 274448f Author: Azeem Shaikh <azeemshaikh38@gmail.com> Date: Tue Mar 14 22:52:30 2023 +0000 Initial implementation of go-git client Signed-off-by: Azeem Shaikh <azeemshaikh38@gmail.com> commit 23bd295 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Mar 14 20:28:41 2023 +0000 :seedling: Bump github/codeql-action from 2.2.4 to 2.2.6 (ossf#2741) commit fc026ef Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Mar 14 17:04:31 2023 +0000 :seedling: Bump github.com/google/ko from 0.12.0 to 0.13.0 in /tools (ossf#2742) commit 2e04214 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Mar 14 14:02:34 2023 +0000 :seedling: Bump tj-actions/changed-files from 35.6.2 to 35.7.0 Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 35.6.2 to 35.7.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](tj-actions/changed-files@5ce975c...bd376fb) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> commit e36b590 Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue Mar 14 08:59:20 2023 -0500 :seedling: Bump actions/cache from 3.3.0 to 3.3.1 (ossf#2740) Bumps [actions/cache](https://github.com/actions/cache) from 3.3.0 to 3.3.1. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@940f3d7...88522ab) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 6ff94eb Author: Gabriela Gutierrez <gabigutierrez@google.com> Date: Mon Mar 13 19:42:37 2023 +0000 :bug: Handle editable pip installs (ossf#2731) * fix: Handle editable pip install Editable pip installs (-e) should be considered secure if the package is installed from a local source or a remote source (VCS install) but pinned by commit hash. To keep the behaviour we have for normal pip installs, we need to guarantee the package dependencies are pinned by hash too. For normal pip installs, we verify that by using --require-hashes flag. Unfortunately, --require-hashes flag is not compatible with editable installs, so we use --no-deps flag to verify the dependencies are not installed since we can't verify if they are pinned. Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * test: Editable pip install in GHA Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * test: Editable pip install in Dockerfile Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * test: Editable pip install in shell script Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * fix: Code complexity increase Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * fix: Simplify boolean return Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * docs: Add pip editable install references in comments Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * fix: Handle multiple packages in editable pip install Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * test: Multi editable pip install in GHA Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * test: Multi editable pip install in Dockerfile Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> * test: Multi editable pip install in shell script Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> --------- Signed-off-by: Gabriela Gutierrez <gabigutierrez@google.com> Co-authored-by: laurentsimon <64505099+laurentsimon@users.noreply.github.com> commit 110e352 Author: raghavkaul <8695110+raghavkaul@users.noreply.github.com> Date: Mon Mar 13 11:13:50 2023 -0400 ✨ Gitlab support: RepoClient (ossf#2655) * Add make targets and E2E test target for GitLab only Signed-off-by: Raghav Kaul <raghavkaul@google.com> * Add GitLab support to RepoClient Signed-off-by: Raghav Kaul <raghavkaul@google.com> * Build * Make target for e2e-gitlab-token * Only run Gitlab tests in CI that don't require a token Signed-off-by: Raghav Kaul <raghavkaul@google.com> * Add tests Signed-off-by: Raghav Kaul <raghavkaul@google.com> * Remove spurious printf Signed-off-by: Raghav Kaul <raghavkaul@google.com> * 🐛 Check OSS Fuzz build file for Fuzzing check (ossf#2719) * Check OSS-Fuzz using project list Signed-off-by: Spencer Schrock <sschrock@google.com> * Use clients.RepoClient interface to perform the new OSS Fuzz check Signed-off-by: Spencer Schrock <sschrock@google.com> * wip: add eager client for better repeated lookup of projects Signed-off-by: Spencer Schrock <sschrock@google.com> * Split lazy and eager behavior into different implementations. Signed-off-by: Spencer Schrock <sschrock@google.com> * Add tests and benchmarks Signed-off-by: Spencer Schrock <sschrock@google.com> * Switch to always parsing JSON to determine if a project is present. The other approach of looking for a substring match would lead to false positives. Signed-off-by: Spencer Schrock <sschrock@google.com> * Add eager constructor to surface status file errors sooner. Signed-off-by: Spencer Schrock <sschrock@google.com> * Switch existing users to new OSS Fuzz client Signed-off-by: Spencer Schrock <sschrock@google.com> * Mark old method as deprecated in the godoc Signed-off-by: Spencer Schrock <sschrock@google.com> * remove unused comment. Signed-off-by: Spencer Schrock <sschrock@google.com> * Use new OSS Fuzz client in e2e test. Signed-off-by: Spencer Schrock <sschrock@google.com> * fix typo. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix potential path bug with test server. Signed-off-by: Spencer Schrock <sschrock@google.com> * Force include the two JSON files which were being ignored by .gitignore Signed-off-by: Spencer Schrock <sschrock@google.com> * trim the status json file Signed-off-by: Spencer Schrock <sschrock@google.com> --------- Signed-off-by: Spencer Schrock <sschrock@google.com> --------- Signed-off-by: Raghav Kaul <raghavkaul@google.com> Signed-off-by: Spencer Schrock <sschrock@google.com> Co-authored-by: Spencer Schrock <sschrock@google.com> commit 5625dda Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Sat Mar 11 17:14:42 2023 +0000 :seedling: Bump github.com/onsi/ginkgo/v2 from 2.8.3 to 2.9.0 in /tools Bumps [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) from 2.8.3 to 2.9.0. - [Release notes](https://github.com/onsi/ginkgo/releases) - [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md) - [Commits](onsi/ginkgo@v2.8.3...v2.9.0) --- updated-dependencies: - dependency-name: github.com/onsi/ginkgo/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> commit d591e38 Author: Spencer Schrock <sschrock@google.com> Date: Fri Mar 10 16:02:05 2023 -0800 🌱 Add RepoClient re-use E2E tests. (ossf#2625) * Add e2e test for re-used repoclient. Signed-off-by: Spencer Schrock <sschrock@google.com> * Improve diff logging Signed-off-by: Spencer Schrock <sschrock@google.com> * Skip scorecard e2e test during unit tests. Signed-off-by: Spencer Schrock <sschrock@google.com> * Fix linter. Signed-off-by: Spencer Schrock <sschrock@google.com> --------- Signed-off-by: Spencer Schrock <sschrock@google.com> commit a7e81bb Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri Mar 10 08:20:28 2023 -0600 :seedling: Bump actions/cache from 3.2.6 to 3.3.0 (ossf#2738) Bumps [actions/cache](https://github.com/actions/cache) from 3.2.6 to 3.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@69d9d44...940f3d7) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

spencerschrock requested review from azeemshaikh38, justaugustus, laurentsimon and naveensrinivasan as code owners January 30, 2023 18:43

spencerschrock had a problem deploying to integration-test January 30, 2023 18:43 — with GitHub Actions Failure

spencerschrock commented Jan 30, 2023

View reviewed changes

laurentsimon approved these changes Jan 30, 2023

View reviewed changes

github-actions bot added the no-pr-activity label Feb 10, 2023

spencerschrock added 4 commits February 10, 2023 12:53

Add e2e test for re-used repoclient.

c4d180c

Signed-off-by: Spencer Schrock <sschrock@google.com>

Improve diff logging

5df724a

Signed-off-by: Spencer Schrock <sschrock@google.com>

Skip scorecard e2e test during unit tests.

8b486cb

Signed-off-by: Spencer Schrock <sschrock@google.com>

Fix linter.

b87025c

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock force-pushed the repo-client-e2e branch from cc22630 to b87025c Compare February 10, 2023 21:36

spencerschrock had a problem deploying to integration-test February 10, 2023 21:36 — with GitHub Actions Failure

github-actions bot removed the no-pr-activity label Feb 11, 2023

github-actions bot added the no-pr-activity label Feb 21, 2023

naveensrinivasan assigned spencerschrock Feb 26, 2023

naveensrinivasan added waiting-for-feedback and removed no-pr-activity labels Feb 26, 2023

Merge branch 'main' into repo-client-e2e

0fab721

spencerschrock requested a review from raghavkaul as a code owner February 28, 2023 01:50

spencerschrock had a problem deploying to integration-test February 28, 2023 01:51 — with GitHub Actions Failure

spencerschrock mentioned this pull request Mar 8, 2023

🐛 Atomically load from accessState to avoid data race. #2732

Merged

2 tasks

Merge branch 'main' into repo-client-e2e

76db929

Signed-off-by: Spencer Schrock <sschrock@google.com>

spencerschrock temporarily deployed to integration-test March 10, 2023 23:49 — with GitHub Actions Inactive

spencerschrock merged commit d591e38 into ossf:main Mar 11, 2023

spencerschrock deleted the repo-client-e2e branch March 11, 2023 00:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Add RepoClient re-use E2E tests. #2625

🌱 Add RepoClient re-use E2E tests. #2625

spencerschrock commented Jan 30, 2023

spencerschrock left a comment

laurentsimon commented Jan 30, 2023

spencerschrock commented Jan 30, 2023

github-actions bot commented Feb 10, 2023

codecov bot commented Feb 10, 2023 •

edited

github-actions bot commented Feb 21, 2023

naveensrinivasan commented Feb 26, 2023

spencerschrock commented Feb 28, 2023

🌱 Add RepoClient re-use E2E tests. #2625

🌱 Add RepoClient re-use E2E tests. #2625

Conversation

spencerschrock commented Jan 30, 2023

What kind of change does this PR introduce?

What is the current behavior?

What is the new behavior (if this is a feature change)?**

Contributors

License

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

spencerschrock left a comment

Choose a reason for hiding this comment

laurentsimon commented Jan 30, 2023

spencerschrock commented Jan 30, 2023

github-actions bot commented Feb 10, 2023

codecov bot commented Feb 10, 2023 • edited

Codecov Report

github-actions bot commented Feb 21, 2023

naveensrinivasan commented Feb 26, 2023

spencerschrock commented Feb 28, 2023

codecov bot commented Feb 10, 2023 •

edited