refactor: Add nightly test stability workflow (aka The Gauntlet) #343
Conversation
| - name: Test with Mock Database | ||
| shell: bash | ||
| env: | ||
| GOCOUNT: 25 |
There was a problem hiding this comment.
you may want to make this a workflow dispatch variable with a default, so when you run things you can tune the count you want: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/
more docs for how to define inputs here: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#providing-inputs
There was a problem hiding this comment.
Found out apparently those inputs don't get populated for pull requests (thread here: https://github.community/t/how-to-set-a-default-value-for-a-workflow-dispatch-variable/156961) - just for workflows, so added a default here too: a0c1800
Codecov Report
@@ Coverage Diff @@
## main #343 +/- ##
==========================================
+ Coverage 67.98% 68.08% +0.09%
==========================================
Files 164 164
Lines 9309 9309
Branches 85 85
==========================================
+ Hits 6329 6338 +9
+ Misses 2353 2344 -9
Partials 627 627
Continue to review full report at Codecov.
|
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| matrix: | ||
| os: |
There was a problem hiding this comment.
you might want to consider adding another variable for a run number, so you can run multiple in parallel (e.g. instance: [1, 2] would let you run 2x each base OS, for a total of 6 runs happening in parallel). since this is happening in the background, the main cost is monetary, and running more instances in parallel will help surface flakes sooner
There was a problem hiding this comment.
Sure, we can try it out. Since this is a nightly job, the performance isn't as important as for a PR gate - so it might be actually be OK to run them serially. Trying it out in: e5c4e18
| env: | ||
| DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }} | ||
| DD_DATABASE: fake | ||
| GIT_COMMIT_MESSAGE: stability-${{ github.event.head_commit.message }} |
There was a problem hiding this comment.
We could use a custom tag instead, that might be better!
It'll require minor edits to the DataDog script I wrote, but that should be simple enough.
There was a problem hiding this comment.
Ah, that sounds good - like maybe an additional trait we could send up?
"test.traits": fmt.Sprintf(`{"database":["%s"]}`, os.Getenv("DD_DATABASE")),
I also noticed we're sending up the workflow:
"ci.pipeline.name": os.Getenv("GITHUB_WORKFLOW"),
That might be sufficient by itself to differentiate the stability run
There was a problem hiding this comment.
Oh ya, it looks like that is available to differentiate and filter these runs:
So I'll remove this superfluous stability- prefix then
There was a problem hiding this comment.
Removed in 14345df
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no activity occurs in the next 5 days. |
|
|
||
| pull_request: | ||
| branches: | ||
| # TODO: Remove before merge |
@kylecarbs has done some awesome work to stamp out and fix intermittent failures to keep our CI stable, fast, and green.
This adds a nightly stability run - where we'll kick off a high-iteration run each night and report to datadog (with a
stability-prefix) to keep an eye on any intermittencies that crop up.