[BUG] [Feature request]: CI - Vitest and github actions with matrix/sharded jobs #5453
Description
Tracer Version(s)
5.36.0
Node.js Version(s)
20.x, 22.x
Vitest Version(s)
3.0.9
Bug Report
Marking as a bug and a feature request as I'm pretty sure it's a new feature, but also can be seen as a bug.
When running larger code bases with a high number of tests, performance can become an issue. In the Vitest documentation there is a way to improve performance by sharding the runs using a github actions matrix to split the tests into nth number of chunked tests. Then a final job to merge the reports and then you can send on your coverage reports etc.
https://vitest.dev/guide/improving-performance.html#sharding
If DD is injected (Using the recommended way) for each sharded job (Setup action + NODE_OPTIONS), they report as individual sessions/modules in Datadog test runs. Additionally when using the --coverage flag the coverage is reported to Datadog as a sharded percentage.
If injecting dd-trace via the recommended NODE_OPTIONS way with vitest's --merge-reports command, then no reporting is sent to Datadog, this makes sense as the --merge-reports flag isn't actually running the tests to trace.
For example, in our codebase at the moment we are running 10 shards to increase the speed of our tests, we currently have ~60% coverage. Each shard can send DD reports, so you end up with 10 sessions each with between 10-15% depending on which subset of tests ran in the shard.
In my view at least, this is where the feature request possibly comes in, it'd be good to setup a "sharded test run" with a variable, then each shard can report it's individual dd-traced performance back without coverage for the tests, then the final --merge-reports --coverage step sends the coverage for the run as a whole. This way in Datadog you still only get one session/run for the workflow as a whole. Ultimately all the shards need to simply be linked back to a singular run.
Reproduction Code
Results in individual shards
name: Unit Test & Coverage
permissions:
contents: read
pull-requests: write
env:
DD_ENV: ci
DD_SERVICE: some-service-name
on:
pull_request:
jobs:
run-tests:
runs-on: ubuntu-latest
continue-on-error: true
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
shardTotal: [10]
steps:
- uses: actions/checkout@v4
- name: Set Node.js 20.x
uses: actions/setup-node@v4
with:
node-version: 20.x
cache: 'yarn'
- name: Configure Datadog Test Optimization
uses: datadog/test-visibility-github-action@v2
with:
languages: js
api_key: ${{secrets.DATADOG_API_KEY}}
site: datadoghq.eu
- name: Install dependencies
run: yarn --frozen-lockfile
- name: Run vitest shards
run: yarn test run --coverage --shard=${{matrix.shardIndex}}/${{matrix.shardTotal}} --reporter=blob --outputFile=./.vitest-reports/report-${{matrix.shardIndex}}.json
env:
NODE_OPTIONS: -r ${{ env.DD_TRACE_PACKAGE }} --import ${{ env.DD_TRACE_ESM_IMPORT }}
- name: Upload blob reports
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: blob-report-${{ matrix.shardIndex }}
path: ./.vitest-reports/*
include-hidden-files: true
retention-days: 1
merge-reports:
if: ${{ !cancelled() }}
needs: [run-tests]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set Node.js 20.x
uses: actions/setup-node@v4
with:
node-version: 20.x
cache: 'yarn'
- name: Install dependencies
run: yarn --frozen-lockfile
- name: Download blob reports from Artifacts
uses: actions/download-artifact@v4
with:
path: .vitest-reports
pattern: blob-report-*
merge-multiple: true
- name: Merge test reports
run: yarn vitest run --silent --merge-reports .vitest-reports --coverage
Results in no-unit test reporting at all
name: Unit Test & Coverage
permissions:
contents: read
pull-requests: write
env:
DD_ENV: ci
DD_SERVICE: some-service-name
on:
pull_request:
jobs:
run-tests:
runs-on: ubuntu-latest
continue-on-error: true
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
shardTotal: [10]
steps:
- uses: actions/checkout@v4
- name: Set Node.js 20.x
uses: actions/setup-node@v4
with:
node-version: 20.x
cache: 'yarn'
- name: Install dependencies
run: yarn --frozen-lockfile
- name: Run vitest shards
run: yarn test run --coverage --shard=${{matrix.shardIndex}}/${{matrix.shardTotal}} --reporter=blob --outputFile=./.vitest-reports/report-${{matrix.shardIndex}}.json
- name: Upload blob reports
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: blob-report-${{ matrix.shardIndex }}
path: ./.vitest-reports/*
include-hidden-files: true
retention-days: 1
merge-reports:
if: ${{ !cancelled() }}
needs: [run-tests]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set Node.js 20.x
uses: actions/setup-node@v4
with:
node-version: 20.x
cache: 'yarn'
- name: Configure Datadog Test Optimization
uses: datadog/test-visibility-github-action@v2
with:
languages: js
api_key: ${{secrets.DATADOG_API_KEY}}
site: datadoghq.eu
- name: Install dependencies
run: yarn --frozen-lockfile
- name: Download blob reports from Artifacts
uses: actions/download-artifact@v4
with:
path: .vitest-reports
pattern: blob-report-*
merge-multiple: true
- name: Merge test reports
run: yarn vitest run --silent --merge-reports .vitest-reports --coverage
env:
NODE_OPTIONS: -r ${{ env.DD_TRACE_PACKAGE }} --import ${{ env.DD_TRACE_ESM_IMPORT }}
Error Logs
No response
Tracer Config
No response
Operating System
No response
Bundling
Unsure