ci: Remove upload of validation reports to Google Cloud #1102

lionel-nj · 2022-02-09T13:19:24Z

Summary:

This PR removes the usage of GCP cloud storage. The content of this PR fixes the behavior described here

Expected behavior:

On forks, acceptance tests should run and not fail because of missing credentials: upload to cloud storage is not used - on this GitHub repository, google cloud storage is used as a backup.

Please make sure these boxes are checked before submitting your pull request - thanks!

~~[ ] Run the unit tests with gradle test to make sure you didn't break anything~~
Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
Linked all relevant issues
Include screenshot(s) showing how this pull request works and fixes the issue(s)

github-actions · 2022-02-09T13:40:25Z

Thank you for this contribution! 🍰✨🦄

Information about source corruption

0 out of 1248 sources are corrupted.

Acceptance test details

The changes in this pull request did not trigger any new errors on known GTFS datasets from the MobilityDatabase.
Download the full acceptance test report for commit 6e94daa here (report will disappear after 90 days).

barbeau

Thanks @lionel-nj! Comments in-line.

barbeau · 2022-02-09T18:37:40Z

.github/workflows/acceptance_test.yml

@@ -186,16 +186,6 @@ jobs:
        with:
          name: reports_all
          path: ${{ github.sha }}/output
-      - name: Set up and authorize Cloud


Could you describe more what we lose by disabling this? Can we still download the reports as linked in the comment generated by the acceptance test (e.g., "Download the full acceptance test report for commit 6e94daa [here] (report will disappear after 90 days)"? If not does that comment need to change?

If it possible to check here if the Action has permission to use the credentials? If so, we could still run this block for branches on this repo, but just not for 3rd party forks.

Could you describe more what we lose by disabling this?

By doing so, we loose the ability to store validation reports on GCP Cloud storage. But it is still possible to download them from the internal storage of github.

Can we still download the reports as linked in the comment generated by the acceptance test (e.g., "Download the full acceptance test report for commit 6e94daa [here] (report will disappear after 90 days)"? If not does that comment need to change?

Yes - this remains possible.

If it possible to check here if the Action has permission to use the credentials? If so, we could still run this block for branches on this repo, but just not for 3rd party forks.

Let me see.

From a quick searchhere and there; it appears that checking the presence of a repo secret in a if block of a job's step might not be possible. However, it might be possible to include an intermediate step that consists in checking the presence of said secret and store the boolean in an environmental variable, then execute Set up and authorize Cloud and Upload reports to Google Cloud Storage conditionally. Unfortunately I will not have the time to finish it 😔.

Looks like this is the issue regarding making execution conditional on secrets - actions/runner#520

barbeau · 2022-02-09T18:38:39Z

Also, I believe the link you have in your original post of "The process of the acceptance tests will solely rely on the internal Github storage in order to avoid the behavior described #755." is pointing to the wrong issue.

github-actions · 2022-02-09T19:59:42Z

Thank you for this contribution! 🍰✨🦄

Information about source corruption

0 out of 1248 sources are corrupted.

Acceptance test details

The changes in this pull request did not trigger any new errors on known GTFS datasets from the MobilityDatabase.
Download the full acceptance test report for commit f632372 here (report will disappear after 90 days).

Based on actions/runner#520 (comment) Expected behavior is that it will upload reports to Google Cloud for PRs for branches on the main repo, but skip upload on PRs from forks.

Based on https://stackoverflow.com/a/68804665/937715

github-actions · 2022-04-27T23:00:40Z

Thank you for this contribution! 🍰✨🦄

Information about source corruption

0 out of 1248 sources are corrupted.

Acceptance test details

The changes in this pull request did not trigger any new errors on known GTFS datasets from the MobilityDatabase.
Download the full acceptance test report for commit 6a9edf6 here (report will disappear after 90 days).

barbeau · 2022-04-28T20:28:17Z

@maximearmstrong @isabelle-dr I've expanded on the work that @lionel-nj started here to effectively skip all validation report uploads to Google Cloud, which should fix acceptance tests on PRs from forks and therefore close #1094.

Ideally we'd conditionally upload reports to Google Cloud only if the PR is for a branch on this repo (and not upload if the PR is from a fork). However, this IF statement in the GitHub Action isn't currently working as expected.

Given that the acceptance tests running on forks is a critical part of the workflow of this repo for evaluating PRs like #1125, and uploading reports to Google Cloud is effectively a "nice to have" feature, I propose that we merge this PR as-is to fix the acceptance tests for forks.

We can then fix the Google Cloud uploads in future work. I've opened issue #1128 to track re-enabling Google Cloud uploads in the future by tweaking the existing code in this PR. I've also marked the code that's not working as intended with a FIXME statement pointing to that issue.

So as long as all the acceptance tests pass for my latest push to this branch, then as far as I know this should be ok to merge if you agree.

github-actions · 2022-04-28T20:34:26Z

Thank you for this contribution! 🍰✨🦄

Information about source corruption

0 out of 1248 sources are corrupted.

Acceptance test details

The changes in this pull request did not trigger any new errors on known GTFS datasets from the MobilityDatabase.
Download the full acceptance test report for commit 79c3021 here (report will disappear after 90 days).

maximearmstrong · 2022-05-03T21:01:16Z

Thank you for this work @barbeau!

@isabelle-dr Do you see value in keeping the reports for more than 90 days? If yes, I agree with Sean that we should merge this PR as-is to fix the problem and then address the Google Cloud issue. If no, I suggest that we remove the code related to Google Cloud right away.

isabelle-dr · 2022-05-04T15:00:38Z

@maximearmstrong @barbeau I'm ok not keeping the report after 90 days, because:

this is what is currently written in the GH action message, so if the user want to keep it for longer, they can
we still have access to the list of datasets in the GH action message "forever", if we need to troubleshoot something that happened in the past

barbeau · 2022-05-04T20:12:03Z

@maximearmstrong @isabelle-dr Sounds good - I've removed the Google Cloud upload code entirely in ced54b9.

After the tests run this PR should be ready for review/merge.

github-actions · 2022-05-04T20:31:16Z

Thank you for this contribution! 🍰✨🦄

Information about source corruption

0 out of 1248 sources are corrupted.

Acceptance test details

The changes in this pull request did not trigger any new errors on known GTFS datasets from the MobilityDatabase.
Download the full acceptance test report for commit 81cd956 here (report will disappear after 90 days).

maximearmstrong

Perfect! Thank you @barbeau and @lionel-nj for this contribution :)

do not use GCP cloud storage

2398996

lionel-nj self-assigned this Feb 9, 2022

lionel-nj marked this pull request as ready for review February 9, 2022 13:44

lionel-nj requested a review from maximearmstrong February 9, 2022 13:44

barbeau reviewed Feb 9, 2022

View reviewed changes

lionel-nj added 2 commits February 9, 2022 15:33

implement secrets-gate

4b963c2

conditionally Upload reports to Google Cloud Storage

142c189

ci: Change format for secrets check

c045d9a

Based on actions/runner#520 (comment) Expected behavior is that it will upload reports to Google Cloud for PRs for branches on the main repo, but skip upload on PRs from forks.

barbeau changed the title ~~ci: do not use GCP cloud storage in acceptance test process~~ ci: Skip upload of validation reports to Google Cloud for PRs from forks Apr 27, 2022

barbeau added 10 commits April 27, 2022 17:50

fix: Use wildcards instead

825dd8d

fix: More formatting changes

f78c86d

fix: Another format change

1deeb70

fix: Formatting

4225ae5

fix: Formatting

1ebe70d

fix: Formatting with -n

e0a5230

fix: Formatting - use ! -z

7e85624

fix: Formatting - double brackets

86474cb

fix: Back to [ -n

d665fe1

fix: Formatting ; then

c729175

Based on https://stackoverflow.com/a/68804665/937715

fix: Invert IF statement

f94dd4b

barbeau changed the title ~~ci: Skip upload of validation reports to Google Cloud for PRs from forks~~ ci: Skip upload of validation reports to Google Cloud Apr 28, 2022

barbeau mentioned this pull request Apr 28, 2022

Upload of acceptance test reports to Google Cloud is always skipped #1128

Closed

docs: Add FIXME

327646b

barbeau requested a review from isabelle-dr April 28, 2022 20:28

barbeau mentioned this pull request Apr 28, 2022

feat: Packaged Executable #1125

Merged

bdferris-v2 mentioned this pull request May 2, 2022

fix: Modify end_to_end.yml to be more specific about which gtfs-validator jar is used #1131

Merged

barbeau changed the title ~~ci: Skip upload of validation reports to Google Cloud~~ ci: Remove upload of validation reports to Google Cloud May 4, 2022

fix: Remove Google Cloud upload entirely

ced54b9

maximearmstrong approved these changes May 4, 2022

View reviewed changes

maximearmstrong merged commit cf170b1 into master May 4, 2022

maximearmstrong deleted the fix/acceptance-test-on-forks branch May 4, 2022 20:43

maximearmstrong mentioned this pull request May 5, 2022

test: allow either order for tooManyCorruptedSource_invalidTest #1091

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Remove upload of validation reports to Google Cloud #1102

ci: Remove upload of validation reports to Google Cloud #1102

lionel-nj commented Feb 9, 2022 •

edited

github-actions bot commented Feb 9, 2022

barbeau left a comment

barbeau Feb 9, 2022

lionel-nj Feb 9, 2022 •

edited by barbeau

lionel-nj Feb 9, 2022 •

edited

barbeau Apr 27, 2022

barbeau commented Feb 9, 2022

github-actions bot commented Feb 9, 2022

github-actions bot commented Apr 27, 2022

barbeau commented Apr 28, 2022

github-actions bot commented Apr 28, 2022

maximearmstrong commented May 3, 2022

isabelle-dr commented May 4, 2022

barbeau commented May 4, 2022

github-actions bot commented May 4, 2022

maximearmstrong left a comment

ci: Remove upload of validation reports to Google Cloud #1102

ci: Remove upload of validation reports to Google Cloud #1102

Conversation

lionel-nj commented Feb 9, 2022 • edited

github-actions bot commented Feb 9, 2022

Information about source corruption

Acceptance test details

barbeau left a comment

Choose a reason for hiding this comment

barbeau Feb 9, 2022

Choose a reason for hiding this comment

lionel-nj Feb 9, 2022 • edited by barbeau

Choose a reason for hiding this comment

lionel-nj Feb 9, 2022 • edited

Choose a reason for hiding this comment

barbeau Apr 27, 2022

Choose a reason for hiding this comment

barbeau commented Feb 9, 2022

github-actions bot commented Feb 9, 2022

Information about source corruption

Acceptance test details

github-actions bot commented Apr 27, 2022

Information about source corruption

Acceptance test details

barbeau commented Apr 28, 2022

github-actions bot commented Apr 28, 2022

Information about source corruption

Acceptance test details

maximearmstrong commented May 3, 2022

isabelle-dr commented May 4, 2022

barbeau commented May 4, 2022

github-actions bot commented May 4, 2022

Information about source corruption

Acceptance test details

maximearmstrong left a comment

Choose a reason for hiding this comment

lionel-nj commented Feb 9, 2022 •

edited

lionel-nj Feb 9, 2022 •

edited by barbeau

lionel-nj Feb 9, 2022 •

edited