Quarantine flaky tests #5631
Quarantine flaky tests #5631
Conversation
schoork
requested review from
compwron,
FireLemons and
littleforest
as code owners
github-actions
bot
added
dependencies
There was a problem hiding this comment.
Nothing actually changed here.
There was a problem hiding this comment.
Nothing actually changed here.
| @@ -84,8 +84,11 @@ group :test do | |||
| gem "capybara-screenshot" | |||
| gem "database_cleaner-active_record" | |||
| gem "email_spec" | |||
| gem "google_drive", "~> 3" | |||
|
|
||
| config.quarantine_database = { | ||
| type: :google_sheets, | ||
| authorization: {type: :service_account_key, file: Rails.root.join("spec", "quarantine", "service-account.json")}, |
There was a problem hiding this comment.
whose account are we going to use here?
|
https://github.com/flexport/quarantine seems unmaintained but probably works |
| - name: Decrypt service-account.json | ||
| run: ./spec/quarantine/decrypt_secret.sh | ||
| env: | ||
| QUARANTINE_SERVICE_ACCOUNT_PASSPHRASE: ${{ secrets.QUARANTINE_SERVICE_ACCOUNT_PASSPHRASE }} |
There was a problem hiding this comment.
I am doing something similar. Unfortunately, this does not work on PRs are pushed from a fork because secrets do not get passed to runner pushed from a fork (https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#using-secrets-in-a-workflow).
Ex: I pulled down this PR and pushed it myself and it failed https://github.com/rubyforgood/casa/actions/runs/8888424582/job/24405217184
I just ran into this issue so I don't have a solution unfortunately.
There was a problem hiding this comment.
It's failing now cause the secret doesn't exist. It could also be the fork issue.
What github issue is this PR for, if any?
N/A
What changed, and why?
Automatically quarantine flaky tests and un-quarantine tests that are no longer flaky. See the wiki for more info.
How will this affect user permissions?
N/A
How is this tested? (please write tests!) 💖💪
Screenshots please :)
Feelings gif (optional)
What gif best describes your feeling working on this issue? https://giphy.com/
How to embed:
Feedback please? (optional)
We are very interested in your feedback! Please give us some :) https://forms.gle/1D5ACNgTs2u9gSdh9