Skip to content

Commit

Permalink
chore(ci): Reduce the number of jobs, move some of them to once a day (
Browse files Browse the repository at this point in the history
…#19001)

* chore(ci): Reduce the number of jobs, move some of them to once a day

Following builds will be done once a day after this is merged:
- all binary engine tests
- json protocol with non-minimal supported version

If PR contains `ci build all` comment, all of those things will be
checked. If not, the job that will test them will run at 9am UTC.

Since we now need two jobs on different schedules, we need a way to
differentiate them: for that reason, both buildpulse and new nightly
jobs are implemented as separate workflows, that call original one
passing the reason (daily/buildpulse) as a parameter.

* Address review comments

* Correctly use inputs

* Update .github/workflows/daily-test.yml

Co-authored-by: Joël Galeran <Jolg42@users.noreply.github.com>

* Rename daily-build to daily-test

* Use free runners for daily builds

* More review feedback

---------

Co-authored-by: Joël Galeran <Jolg42@users.noreply.github.com>
  • Loading branch information
SevInf and Jolg42 committed May 5, 2023
1 parent 0339b08 commit 686d4d1
Show file tree
Hide file tree
Showing 5 changed files with 136 additions and 28 deletions.
15 changes: 15 additions & 0 deletions .github/workflows/daily-buildpulse.yml
@@ -0,0 +1,15 @@
name: Daily BuildPulse run

on:
workflow_dispatch:
schedule:
# https://crontab.guru - “At every 30th minute past every hour from 3 through 6.”
- cron: '*/30 3-6 * * *'

jobs:
run_tests:
uses: ./.github/workflows/test.yml
with:
reason: buildpulse
jobTimeout: 70
ubuntuRunner: ubuntu-latest
32 changes: 32 additions & 0 deletions .github/workflows/daily-test.yml
@@ -0,0 +1,32 @@
name: Daily Test

on:
workflow_dispatch:
schedule:
# 4am daily Mon-Fri
- cron: '0 4 * * 1-5'

jobs:
run_tests:
uses: ./.github/workflows/test.yml
with:
reason: daily-test
jobTimeout: 70
ubuntuRunner: ubuntu-latest
failure:
needs:
- run_tests
if: ${{ failure() }}
name: Communicate failure
runs-on: ubuntu-latest
steps:
- name: Set current job url in SLACK_FOOTER env var
run: echo "SLACK_FOOTER=<$GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID|Click here to go to the job logs>" >> $GITHUB_ENV

- name: Slack Notification on Failure
uses: rtCamp/action-slack-notify@v2.2.0
env:
SLACK_TITLE: 'prisma/prisma daily build failed :x:'
SLACK_COLOR: '#FF0000'
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
SLACK_CHANNEL: feed-prisma-daily-build-failures
30 changes: 30 additions & 0 deletions .github/workflows/scripts/build-client-matrix.js
@@ -0,0 +1,30 @@
#!/usr/bin/env node
'use strict'
const fs = require('fs')

const [reason, buildComment] = process.argv.slice(2)
console.log({ reason, buildComment })

const hasBuildAllComment = buildComment?.length > 0

const excludeClient = []
const excludeDataProxy = []

const queryEngine = ['library']

if (hasBuildAllComment || reason === 'daily-test') {
queryEngine.push('binary')
} else {
excludeClient.push({ node: 18, engineProtocol: 'json' })
excludeDataProxy.push({ node: 16, engineProtocol: 'json' }, { node: 18, engineProtocol: 'json' })
}

const outputRaw = { queryEngine, excludeClient, excludeDataProxy }

const outputsStr = Object.entries(outputRaw)
.map(([key, value]) => `${key}=${JSON.stringify(value)}`)
.join('\n')

console.log(outputsStr)

fs.appendFileSync(process.env.GITHUB_OUTPUT, outputsStr, 'utf8')
84 changes: 56 additions & 28 deletions .github/workflows/test.yml
Expand Up @@ -42,10 +42,24 @@ on:
- '*.bench.ts'
- 'scripts/ci/publish.ts'
- 'graphs/**'
schedule:
# https://crontab.guru - “At every 30th minute past every hour from 3 through 6.”
- cron: '*/30 3-6 * * *'
workflow_dispatch:
workflow_call:
inputs:
reason:
description: |
Reason why this workflow was called. Could be "buildpulse" for
flaky tests detection or "daily-test" for scheduled full builds
type: string
required: false
default: CI
ubuntuRunner:
description: Override default runner used for linux tests. Can be used to opt-out of buildjet
type: string
required: false
jobTimeout:
description: Timeout for the jobs. Default is 35 minutes
type: number
required: false

env:
HAS_BUILDPULSE_SECRETS: ${{ secrets.BUILDPULSE_ACCESS_KEY_ID != '' && secrets.BUILDPULSE_SECRET_ACCESS_KEY != '' }}
Expand All @@ -55,15 +69,18 @@ env:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
# Don't cancel if it's on schedule (used for BuildPulse for flaky tests detection)
cancel-in-progress: ${{ github.event_name != 'schedule' }}
# Don't cancel if called from buildpulse workflow
cancel-in-progress: ${{ inputs.reason != 'buildpulse' }}

jobs:
detect_jobs_to_run:
name: Detect jobs to run
runs-on: ubuntu-latest
outputs:
jobs: ${{ steps.detect.outputs.jobs }}
queryEngine: ${{ steps.determineBuildMatrixParams.outputs.queryEngine }}
excludeClient: ${{ steps.determineBuildMatrixParams.outputs.excludeClient }}
excludeDataProxy: ${{ steps.determineBuildMatrixParams.outputs.excludeDataProxy }}
steps:
- id: checkout
uses: actions/checkout@v3
Expand All @@ -74,6 +91,15 @@ jobs:
continue-on-error: true # This allow this job to fail (e.g. for scheduled runs), and then the `detect` step below will default to `all`
- id: detect
run: ./.github/workflows/scripts/detect-jobs-to-run.js <<< '${{ steps.files.outputs.all }}'
- name: Find "ci test all" comment
uses: peter-evans/find-comment@v2
id: findTestAllComment
if: ${{ github.event_name == 'pull_request' }}
with:
issue-number: ${{ github.event.pull_request.number }}
body-includes: 'ci test all'
- id: determineBuildMatrixParams
run: ./.github/workflows/scripts/build-client-matrix.js '${{ inputs.reason }}' '${{ steps.findTestAllComment.outputs.comment-id }}'

#
# Linting
Expand Down Expand Up @@ -115,19 +141,20 @@ jobs:
# CLIENT (without types test)
#
client:
timeout-minutes: ${{ github.event_name == 'schedule' && 70 || 35 }}
runs-on: ${{ github.event_name == 'schedule' && contains(matrix.os, 'buildjet') && 'ubuntu-latest' || matrix.os }}
timeout-minutes: ${{ inputs.jobTimeout || 35 }}
runs-on: ${{ inputs.ubuntuRunner || matrix.os }}

needs: detect_jobs_to_run
if: ${{ contains(needs.detect_jobs_to_run.outputs.jobs, '-all-') || contains(needs.detect_jobs_to_run.outputs.jobs, '-client-') }}

strategy:
fail-fast: false
matrix:
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}
engineProtocol: ['graphql', 'json']
os: [buildjet-4vcpu-ubuntu-2004]
node: [16, 18]
exclude: ${{ fromJson(needs.detect_jobs_to_run.outputs.excludeClient) }}

env:
CI: true
Expand Down Expand Up @@ -210,7 +237,7 @@ jobs:
- name: Upload test results to BuildPulse for flaky test detection
# Only run this step for branches where we have access to secrets.
# Run this step even when the tests fail. Skip if the workflow is cancelled.
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && github.event_name == 'schedule'
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && inputs.reason == 'buildpulse'
uses: Workshop64/buildpulse-action@main
with:
account: 17219288
Expand All @@ -220,7 +247,7 @@ jobs:
secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

- uses: actions/upload-artifact@v3
if: github.event_name == 'schedule'
if: inputs.reason == 'buildpulse'
with:
name: ${{ github.job }}_${{ matrix.os }}_node-${{ matrix.node }}_dataproxy_junit.xml
path: packages/*/junit*.xml
Expand All @@ -229,8 +256,8 @@ jobs:
# CLIENT (functional tests with mini-proxy)
#
client-dataproxy:
timeout-minutes: ${{ github.event_name == 'schedule' && 70 || 35 }}
runs-on: ${{ github.event_name == 'schedule' && contains(matrix.os, 'buildjet') && 'ubuntu-latest' || matrix.os }}
timeout-minutes: ${{ inputs.jobTimeout || 35 }}
runs-on: ${{ inputs.ubuntuRunner || matrix.os }}

needs: detect_jobs_to_run
if: ${{ contains(needs.detect_jobs_to_run.outputs.jobs, '-all-') || contains(needs.detect_jobs_to_run.outputs.jobs, '-client-') }}
Expand All @@ -241,6 +268,7 @@ jobs:
os: [buildjet-4vcpu-ubuntu-2004]
node: [14, 16, 18]
engineProtocol: ['graphql', 'json']
exclude: ${{ fromJson(needs.detect_jobs_to_run.outputs.excludeDataProxy) }}

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -315,7 +343,7 @@ jobs:
- name: Upload test results to BuildPulse for flaky test detection
# Only run this step for branches where we have access to secrets.
# Run this step even when the tests fail. Skip if the workflow is cancelled.
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && github.event_name == 'schedule'
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && inputs.reason == 'buildpulse'
uses: Workshop64/buildpulse-action@main
with:
account: 17219288
Expand All @@ -325,7 +353,7 @@ jobs:
secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

- uses: actions/upload-artifact@v3
if: github.event_name == 'schedule'
if: inputs.reason == 'buildpulse'
with:
name: ${{ github.job }}_${{ matrix.os }}_node-${{ matrix.node }}_dataproxy_junit.xml
path: packages/*/junit*.xml
Expand All @@ -334,8 +362,8 @@ jobs:
# CLIENT (memory tests)
#
client-memory:
timeout-minutes: ${{ github.event_name == 'schedule' && 30 || 15 }}
runs-on: ${{ github.event_name == 'schedule' && contains(matrix.os, 'buildjet') && 'ubuntu-latest' || matrix.os }}
timeout-minutes: ${{ inputs.jobTimeout || 35 }}
runs-on: ${{ inputs.ubuntuRunner || matrix.os }}

needs: detect_jobs_to_run
if: ${{ contains(needs.detect_jobs_to_run.outputs.jobs, '-all-') || contains(needs.detect_jobs_to_run.outputs.jobs, '-client-') }}
Expand Down Expand Up @@ -578,7 +606,7 @@ jobs:
- name: Upload test results to BuildPulse for flaky test detection
# Only run this step for branches where we have access to secrets.
# Run this step even when the tests fail. Skip if the workflow is cancelled.
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && github.event_name == 'schedule'
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && inputs.reason == 'buildpulse'
uses: Workshop64/buildpulse-action@main
with:
account: 17219288
Expand All @@ -588,7 +616,7 @@ jobs:
secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

- uses: actions/upload-artifact@v3
if: github.event_name == 'schedule'
if: inputs.reason == 'buildpulse'
with:
name: ${{ github.job }}_${{ matrix.os }}_node-${{ matrix.node }}_${{ matrix.queryEngine }}_junit.xml
path: packages/*/junit*.xml
Expand Down Expand Up @@ -652,7 +680,7 @@ jobs:
strategy:
fail-fast: false
matrix:
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}
node: [16]

steps:
Expand Down Expand Up @@ -726,7 +754,7 @@ jobs:
strategy:
fail-fast: false
matrix:
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}
os: [ubuntu-latest]
node: [16, 18]

Expand Down Expand Up @@ -798,7 +826,7 @@ jobs:
strategy:
fail-fast: false
matrix:
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}
os: [ubuntu-latest]
node: [16, 18]

Expand Down Expand Up @@ -880,7 +908,7 @@ jobs:
strategy:
fail-fast: false
matrix:
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}
os: [ubuntu-latest]
node: [16, 18]

Expand Down Expand Up @@ -1065,7 +1093,7 @@ jobs:
matrix:
os: [macos-latest, windows-latest]
node: [14]
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}
# Commented out as we still have limited capacity for macOS runners
# When we have more capacity we can shard again
# shard: ['1', '2']
Expand Down Expand Up @@ -1137,7 +1165,7 @@ jobs:
- name: Upload test results to BuildPulse for flaky test detection
# Only run this step for branches where we have access to secrets.
# Run this step even when the tests fail. Skip if the workflow is cancelled.
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && github.event_name == 'schedule'
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && inputs.reason =='buildpulse'
uses: Workshop64/buildpulse-action@main
with:
account: 17219288
Expand All @@ -1147,7 +1175,7 @@ jobs:
secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

- uses: actions/upload-artifact@v3
if: github.event_name == 'schedule'
if: inputs.reason == 'buildpulse'
with:
name: ${{ github.job }}_${{ matrix.os }}_node-${{ matrix.node }}_${{ matrix.queryEngine }}_junit.xml
path: packages/*/junit*.xml
Expand All @@ -1171,7 +1199,7 @@ jobs:
matrix:
os: [macos-latest, windows-latest]
node: [14]
queryEngine: ['library', 'binary']
queryEngine: ${{ fromJson(needs.detect_jobs_to_run.outputs.queryEngine) }}

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -1376,7 +1404,7 @@ jobs:
- name: Upload test results to BuildPulse for flaky test detection
# Only run this step for branches where we have access to secrets.
# Run this step even when the tests fail. Skip if the workflow is cancelled.
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && github.event_name == 'schedule'
if: env.HAS_BUILDPULSE_SECRETS == 'true' && !cancelled() && inputs.reason == 'buildpulse'
uses: Workshop64/buildpulse-action@main
with:
account: 17219288
Expand All @@ -1386,7 +1414,7 @@ jobs:
secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}

- uses: actions/upload-artifact@v3
if: github.event_name == 'schedule'
if: inputs.reason == 'buildpulse'
with:
name: ${{ github.job }}_${{ matrix.os }}_node-${{ matrix.node }}_${{ matrix.queryEngine }}_junit.xml
path: packages/*/junit*.xml
3 changes: 3 additions & 0 deletions TESTING.md
Expand Up @@ -503,6 +503,9 @@ By creating a Pull Request the following pipelines will be triggered

They are both running the same tests but with different Node.js version and will need to be successful before merging ("flaky" tests might show up and might be ignored).

By default, some of the node version/engine/engine protocol combination are tested only
during daily builds. If you need to run all of them for your PR leave `ci test all` comment on the PR and re-run the workflow.

### Publishing an integration version of all the packages

If a branch name starts with `integration/` like `integration/fix-all-the-things` the [Buildkite `[Release] Prisma TypeScript`](https://buildkite.com/prisma/release-prisma-typescript) pipeline will be triggered.
Expand Down

0 comments on commit 686d4d1

Please sign in to comment.