Skip to content

chore(github): add CodSpeed workflow #6785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
benjamincanac merged 4 commits into from
Jul 31, 2026
Merged

chore(github): add CodSpeed workflow #6785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
899a1df
ci: add CodSpeed benchmark workflow
benjamincanac Jul 31, 2026
bf762c8
ci: run benchmarks from module.yml
benjamincanac Jul 31, 2026
ea4efc1
ci: skip benchmarks on forks
benjamincanac Jul 31, 2026
6b2cd9b
test(bench): mount re-render wrappers inside the bench function
benjamincanac Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/module.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,3 +190,43 @@ jobs:

- name: Typecheck
run: pnpm run typecheck

benchmarks:
runs-on: ubuntu-latest

# Skip forks of the repo, they can't authenticate with CodSpeed
if: github.repository_owner == 'nuxt'

permissions:
contents: read
id-token: write # for OpenID Connect authentication with CodSpeed

env:
NUXT_GITHUB_TOKEN: ${{ secrets.NUXT_GITHUB_TOKEN }}

steps:
- name: Checkout
uses: actions/checkout@v7
with:
persist-credentials: false

- name: Install pnpm
uses: pnpm/action-setup@v6

- name: Install node
uses: actions/setup-node@v7
with:
node-version: 24
cache: pnpm

- name: Install dependencies
run: pnpm install

- name: Prepare
run: pnpm run dev:prepare

- name: Run benchmarks
uses: CodSpeedHQ/action@v4
with:
mode: simulation
run: pnpm run bench
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@
"vue-component-type-helpers": "^3.3.8"
},
"devDependencies": {
"@codspeed/vitest-plugin": "^5.7.1",
"@nuxt/eslint-config": "^1.16.0",
"@nuxt/module-builder": "^1.0.3",
"@nuxt/test-utils": "^4.1.0",
Expand Down
228 changes: 218 additions & 10 deletions pnpm-lock.yaml
View file
Open in desktop

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions pnpm-workspace.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ ignoreWorkspaceRootCheck: true
linkWorkspacePackages: true

overrides:
# `@codspeed/vitest-plugin` caps its `vite` peer at ^7 while the repo resolves
# vite 8 for the Nuxt modules; without widening it, pnpm's peer dedupe re-keys
# the whole vite peer group down to vitest's vite 7. The plugin only ever runs
# inside vitest's own vite, so the wider range is bookkeeping, not runtime.
"@codspeed/vitest-plugin>vite": ^7.0.0 || ^8.0.0
"@nuxt/kit": ^4.5.1
"@nuxt/ui": workspace:*
unplugin: ^3.3.0
Expand Down
12 changes: 4 additions & 8 deletions test/bench/button-link.bench.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,18 +33,14 @@ describe('mount', () => {
// ends in the initial state), re-rendering the whole subtree.
function reRenderBench(name: string, comp: any, props: Record<string, any> = {}) {
describe(`re-render: ${name}`, () => {
let wrapper: Awaited<ReturnType<typeof mountSuspended>>
let wrapper: Awaited<ReturnType<typeof mountSuspended>> | undefined

// Mounted lazily on the first call: CodSpeed's analysis runner invokes the
// bench function without tinybench's `setup`/`teardown` options.
bench(name, async () => {
wrapper ??= await mountSuspended(comp, { props })
await wrapper.setProps({ loading: true })
await wrapper.setProps({ loading: false })
}, {
async setup() {
wrapper = await mountSuspended(comp, { props })
},
teardown() {
wrapper?.unmount()
}
})
})
}
Comment on lines 34 to 46

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

CodSpeed vitest-plugin walltime simulation mode warmup or setup invocation behavior

💡 Result:

When using CodSpeed with Vitest, the instrumentation and measurement modes are primarily handled by the CodSpeed CI environment and the @codspeed/vitest-plugin [1][2]. The invocation behavior for setup and warmup varies based on the measurement mode and the specific configuration [3][4]. 1. Warmup and Setup Invocation CodSpeed manages the lifecycle of your benchmarks to ensure measurement consistency [5][3]. While you define your benchmarks using standard Vitest bench syntax, CodSpeed handles the warmup and repetition cycles automatically in the CI environment [5][3]. For complex setups (such as database connections or external services), you can use Vitest's setupFiles configuration or standard lifecycle hooks like beforeAll, optionally combined with the setupInstruments utility provided by the @codspeed/vitest-plugin if specific instrumentation initialization is required [6]. 2. Simulation vs. Walltime Modes The invocation behavior and underlying measurement approach differ significantly between modes: - CPU Simulation (Default): This mode measures performance by simulating CPU cycles [3][7]. It is hardware-agnostic, deterministic, and typically produces results with <1% variance [3][7]. It does not measure system calls or I/O-bound operations and is best suited for CPU-intensive tasks [5][3][7]. - Walltime Mode: This mode measures the actual "wall clock" elapsed time, capturing system calls, I/O, and multi-threading effects [5][3][4]. Because it is sensitive to environmental noise, it requires consistent hardware [5][4]. CodSpeed recommends using their managed "codspeed-macro" runners when executing in this mode to ensure high precision [8][5][4]. 3. Invocation in CI To enable these modes, you specify the desired mode in the CodSpeed GitHub Action [9][8]. When running in CI, the CodSpeed plugin detects the instrumented environment and replaces or enhances the default Vitest runner [1][2][10]. If you run the command locally or outside of a supported CI environment, the plugin falls back to the default Vitest runner, and CodSpeed-specific instrumentation is not applied [1][2][10]. In summary, you do not manually invoke warmup or simulation logic; instead, you configure the mode via the CodSpeed GitHub Action and rely on the @codspeed/vitest-plugin to manage the execution lifecycle during your CI runs [9][8][3]. If you have tasks that require specific preparation, standard Vitest lifecycle mechanisms are the appropriate way to handle them [6].

Citations:

🌐 Web query:

CodSpeed vitest-plugin analysis runner setup teardown bench invocation order

💡 Result:

The CodSpeed vitest-plugin operates by instrumenting your benchmarks to capture performance metrics accurately, specifically isolating the measured code execution from framework overhead [1]. Regarding the invocation order and lifecycle, the plugin is designed to bracket the measurement window specifically around the benchmarked loop [1]. This ensures that setup and teardown hooks—whether provided by the Vitest bench environment or user-defined logic—are typically excluded from the recorded walltime measurements [1]. Key points regarding the runner, setup, and teardown behavior include: 1. Instrumentation Scope: The CodSpeed runner wraps the measured function within a root frame, specifically ensuring that internal setup/teardown hooks (and post-loop statistical computations performed by the underlying runner, such as tinybench) fall outside the measured sample window [1]. 2. Marker Protocol: The plugin employs a specific marker sequence—SampleStart > BenchmarkStart > BenchmarkEnd > SampleEnd—to ensure that instrumentation remains balanced even if a benchmark throws an error [2]. This is handled via try/finally blocks within the runner [2]. 3. Execution Environment: When run locally, the plugin acts as a no-op (falling back to the default Vitest runner) [3][4][5]. The CodSpeed instrumentation only activates in CI environments where performance data capture is required [3][5]. 4. Setup/Teardown Handling: Because CodSpeed integrates with Vitest's existing lifecycle, standard Vitest setup/teardown (like beforeEach/afterEach) is respected, but CodSpeed’s instrumentation is strategically placed to avoid including the timing of these hooks in the final benchmark metrics [1]. For users requiring fine-grained control over setup/teardown outside of standard Vitest hooks, CodSpeed typically recommends the pattern used in its other framework integrations (such as the pedantic mode in pytest-codspeed) [6][7], though within Vitest, you should rely on the plugin's automatic management of the measurement window to ensure accurate results [1]. If you are encountering issues with setup/teardown being called multiple times, ensure the plugin is applied only to your benchmarking project in your vitest.config.ts, rather than globally, to avoid duplicate teardown events across multiple test projects [3].

Citations:

🌐 Web query:

CodSpeed vitest plugin source runner setup teardown call bench

💡 Result:

The @codspeed/vitest-plugin integrates directly with Vitest's existing benchmarking capabilities, meaning you do not need special CodSpeed-specific calls for setup and teardown [1][2]. Instead, you leverage standard Vitest hooks and benchmarking patterns, which CodSpeed then instruments automatically when run in a supported CI environment [1][2]. To set up CodSpeed with Vitest: 1. Installation: Install the plugin and Vitest (v3.2+) as dev dependencies [1][2][3]. npm install --save-dev @codspeed/vitest-plugin vitest 2. Configuration: Add the plugin to your vitest.config.ts (or.mts) file [2][4]: import { defineConfig } from "vitest/config"; import codspeedPlugin from "@codspeed/vitest-plugin"; export default defineConfig({ plugins: [codspeedPlugin], }); 3. Writing Benchmarks: Continue using the standard Vitest bench function and standard test hooks like beforeEach, afterEach, beforeAll, and afterAll [2][4]. CodSpeed intercepts these processes during CI execution to provide performance instrumentation [1][2]. Example structure: import { describe, bench, beforeAll, afterAll } from "vitest"; describe("my-benchmark", => { beforeAll(async => { // Standard Vitest setup }); afterAll(async => { // Standard Vitest teardown }); bench("my-task", => { // Perform task }); }); When running locally, CodSpeed detects the environment and gracefully falls back to the default Vitest runner, outputting standard benchmark results [1][2][5]. CodSpeed instrumentation is specifically intended for CI pipelines to ensure consistent, reliable performance measurements [2][3].

Citations:

🌐 Web query:

CodSpeed walltime runner bench callback invoked multiple times simulation mode

💡 Result:

The behavior you are observing is a fundamental difference between CodSpeed's two primary measurement instruments: Simulation and Walltime [1]. In CodSpeed, benchmarks are invoked differently depending on the measurement mode [1][2]: 1. Simulation Mode (Default): This instrument simulates CPU behavior to provide consistent, hardware-agnostic measurements [1][3]. In this mode, each benchmark is designed to run only once [3][4][5]. Because it measures simulated CPU cycles and cache/memory access patterns rather than real-time duration, multiple invocations are unnecessary and are typically avoided by the runner [1][3]. 2. Walltime Mode: This instrument measures the actual "wall-clock" time of your benchmark [6][7]. To ensure statistical significance and handle real-world noise (such as I/O or background processes), the CodSpeed runner intentionally invokes the benchmark callback multiple times [6][2]. The process typically involves [2]: - A Warmup Phase: The benchmark runs repeatedly until a steady state is reached [2]. - A Measurement Phase: Based on the warmup results, the runner calculates the number of rounds required to satisfy configured time or round constraints, executing the benchmark multiple times to collect timing data for each round [2]. If you see your benchmark callback being invoked multiple times, it is confirmed that you are running in (or have configured) Walltime mode [2]. If you require your benchmark to run only once, you must ensure you are using the Simulation instrument [3][4]. You can control this via: - The mode parameter in the CodSpeed GitHub Action (e.g., mode: "simulation") [8]. - The CODSPEED_RUNNER_MODE environment variable [9][10]. - CLI flags like --measurement-mode simulation [11][5] or C++ CMake flags [4]. Note that parallel execution of multiple benchmark processes is discouraged in Walltime mode as it introduces noise and reduces measurement precision [6][7].

Citations:

🌐 Web query:

CodSpeed vitest plugin runner.js mountSuspended bench invocation CodSpeed

💡 Result:

The CodSpeed Vitest plugin and the mountSuspended utility are two distinct tools used in different contexts within the Vitest ecosystem. The @codspeed/vitest-plugin is used to instrument and run performance benchmarks [1][2]. To use it, you configure it as a plugin in your vitest.config.ts file [1][2]: import { defineConfig } from "vitest/config"; import codspeedPlugin from "@codspeed/vitest-plugin"; export default defineConfig({ plugins: [codspeedPlugin], }); You then run your benchmarks using the standard vitest bench command [1][2]. Locally, CodSpeed will fall back to the default Vitest runner, while in CI environments, it provides instrumented performance measurements [1][3]. The mountSuspended utility, on the other hand, is a specific tool from @nuxt/test-utils used to mount Vue components within a Nuxt environment during unit tests [4][5]. It allows for async setup and access to Nuxt context (such as plugins and auto-imports) [4][5]. It is not a benchmarking tool and is not typically used inside a CodSpeed bench invocation [4][6]. There is no direct integration between mountSuspended and the CodSpeed runner. If you intend to benchmark a component that requires mountSuspended (e.g., measuring the performance of a Nuxt component setup), you would likely need to manually manage the Nuxt environment within the bench task, although this is generally not the recommended pattern for performance benchmarking, which focuses on stable, isolated logic execution [2][5]. Performance benchmarks in CodSpeed are designed to measure code execution time, whereas mountSuspended is designed for functional testing within a specific framework lifecycle [2][5].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== package files =="
git ls-files | rg '(^package\.json$|pnpm-lock\.yaml$|package-lock\.json$|yarn\.lock$)' | sed -n '1,50p'

echo
echo "== CodSpeed/tinybench references =="
rg -n "`@codspeed`|codspeed|tinybench|CODSPEED|measurement-mode|mode:" -S . \
  --glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' | sed -n '1,120p'

echo
echo "== relevant bench files snippets =="
for f in test/bench/button-link.bench.ts test/bench/components.bench.ts test/bench/link-stack.bench.ts; do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    sed -n '1,110p' "$f" | cat -n
  fi
done

echo
echo "== vitest config and script snippets =="
if [ -f vitest.config.ts ]; then sed -n '1,200p' vitest.config.ts | cat -n; fi
if [ -f package.json ]; then jq '.scripts, .devDependencies, .dependencies' package.json | sed -n '1,220p'; fi

Repository: nuxt/ui

Length of output: 25170

🌐 Web query:

CodSpeed vitest plugin 5.7.1 benchmarks runner invocations bench callback setup teardown walltime simulation

💡 Result:

The CodSpeed vitest plugin facilitates benchmarking by integrating with the Vitest test runner. It requires Vitest v3.2 or higher [1][2]. While it allows developers to write benchmarks using the standard vitest bench function [1][3], the plugin specifically handles the instrumentation required to export these results to the CodSpeed platform for performance analysis in CI/CD environments [1][2]. Regarding runner invocations and performance measurement: Runner Invocations and Hooks: The CodSpeed vitest plugin maintains compatibility with Vitest's benchmark hooks, such as setup and teardown, while ensuring that performance metrics are captured accurately [4]. Recent optimizations have been implemented to ensure that the walltime measurement window is strictly bracketed around the measured loop itself [4]. This prevents framework overhead—such as the execution of setup/teardown hooks or post-loop result computation—from polluting the collected metrics, which previously resulted in inaccurate flamegraphs [4]. Instruments: CodSpeed supports two primary measurement instruments [5]: 1. CPU Simulation: This is the recommended default for most benchmarks [5]. It simulates CPU behavior to provide consistent, hardware-agnostic results regardless of system load [6][5]. Benchmarks using this mode typically run once, making it unsuitable for highly I/O-intensive code where simulation overhead may be significant [5]. 2. Walltime: This instrument measures real-world elapsed time [6][7]. It is best suited for benchmarks that involve significant I/O, network calls, or multi-threading, as it captures the actual performance observed in a real environment [8][5]. When using the walltime instrument, CodSpeed typically recommends using their bare-metal "Macro Runners" to ensure high precision and low noise in the measurements [7]. Locally, when running vitest bench without the CodSpeed CI instrumentation, the plugin gracefully falls back to the default Vitest runner, allowing developers to verify their benchmarks without requiring the full CodSpeed profiling infrastructure [1][2].

Citations:

🌐 Web query:

CodSpeed github runner walltime simulation bench invocations setup teardown tinybench

💡 Result:

CodSpeed provides two primary modes of measurement for performance benchmarking in CI: CPU simulation (default/recommended) and walltime [1][2]. Walltime Measurement and Macro Runners Walltime measurement captures the actual elapsed (wall-clock) time, including I/O operations, system calls, and multi-threaded execution effects [3][4]. Because traditional shared cloud runners introduce significant performance noise that can distort walltime data, CodSpeed requires the use of CodSpeed Macro Runners for this mode [5][6]. These are dedicated, 16-core ARM64 bare-metal machines managed by CodSpeed that provide a stable, isolated environment [5]. To enable walltime in a GitHub Actions workflow, you must: 1. Change the runner to use the CodSpeed-managed environment by setting runs-on: codspeed-macro [7][5]. 2. Configure the CodSpeed Action to use the walltime instrument by setting mode: walltime [7][1]. Tinybench Integration CodSpeed integrates with tinybench via the @codspeed/tinybench-plugin [8][9]. This plugin allows you to wrap your tinybench suite, enabling CodSpeed instrumentation [8]. Key aspects of the integration include: - Setup/Teardown Handling: The plugin is designed to exclude framework overhead from the measurements. It specifically targets the measured loop, ensuring that run-mode setup and teardown hooks (as well as statistical computation post-loop) are bracketed outside the walltime instrumentation window [10]. - Instrumentation Window: Recent updates to the plugin ensure that benchmark markers are emitted inside the sample window to maintain proper nesting (SampleStart > BenchmarkStart > BenchmarkEnd > SampleEnd), which is critical for accurate reporting [11]. - Versioning: The plugin supports multiple versions of tinybench (v4+), with internal adjustments to handle changes in tinybench's private field layouts and option accessibility in newer versions [8][12]. - Local Fallback: When run outside of the CI environment or without the full CodSpeed CLI/action instrumentation, the plugin automatically falls back to standard tinybench execution [8][9]. For optimal consistency, it is recommended to ensure your cache keys include ${{ runner.arch }} because Macro Runners use the ARM64 architecture [7][5].

Citations:

Do not rely on a single-run CodSpeed invocation for lazy mounts. CodSpeed simulation/bench runners invoke the benchmark function multiple times per sample loop, so wrapper ??= await mountSuspended(...) can include the one-time mountSuspended cost in measured re-render timings. Keep component re-render benchmarks benching the toggle cycle over an already mounted component, and run mountSuspended separately as its own benchmark.

📍 Affects 3 files
  • test/bench/button-link.bench.ts#L34-L46 (this comment)
  • test/bench/components.bench.ts#L26-L34
  • test/bench/components.bench.ts#L73-L77
  • test/bench/link-stack.bench.ts#L48-L53
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/bench/button-link.bench.ts` around lines 34 - 46, Separate initial
mounting from re-render measurement in reRenderBench and the corresponding sites
in test/bench/components.bench.ts lines 26-34 and 73-77 and
test/bench/link-stack.bench.ts lines 48-53: add dedicated mount benchmarks,
ensure each re-render benchmark starts with an already mounted component, and
measure only the loading toggle cycle without lazy mount logic. The anchor site
is test/bench/button-link.bench.ts lines 34-46.

Expand Down
23 changes: 7 additions & 16 deletions test/bench/components.bench.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,15 @@ describe('Button mount', () => {
// its subtree. Each iteration performs a full on/off cycle so the work measured
// is deterministic and always ends back in the initial `false` state.
describe('Button re-render (variant prop change)', () => {
let wrapper: Awaited<ReturnType<typeof mountSuspended>>
let wrapper: Awaited<ReturnType<typeof mountSuspended>> | undefined

// Mounted lazily on the first call: CodSpeed's analysis runner invokes the
// bench function without tinybench's `setup`/`teardown` options, so a mount
// there never happens under instrumentation. The warmup pass absorbs it.
bench('toggle loading', async () => {
wrapper ??= await mountSuspended(Button, { props: { label: 'Button' } })
await wrapper.setProps({ loading: true })
await wrapper.setProps({ loading: false })
}, {
async setup() {
wrapper = await mountSuspended(Button, { props: { label: 'Button' } })
},
teardown() {
wrapper?.unmount()
}
})
})

Expand Down Expand Up @@ -73,16 +70,10 @@ describe('Table mount (200 x 5)', () => {
})

describe('Table re-render (new data identity)', () => {
let wrapper: Awaited<ReturnType<typeof mountSuspended>>
let wrapper: Awaited<ReturnType<typeof mountSuspended>> | undefined

bench('set data', async () => {
wrapper ??= await mountSuspended(Table, { props: tableProps(makeData(200)) })
await wrapper.setProps({ data: makeData(200) })
}, {
async setup() {
wrapper = await mountSuspended(Table, { props: tableProps(makeData(200)) })
},
teardown() {
wrapper?.unmount()
}
})
})
12 changes: 4 additions & 8 deletions test/bench/link-stack.bench.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,18 +45,14 @@ describe('mount', () => {
describe('re-render', () => {
for (const [name, render] of CASES) {
describe(name, () => {
let wrapper: Awaited<ReturnType<typeof mountSuspended>>
let wrapper: Awaited<ReturnType<typeof mountSuspended>> | undefined

// Mounted lazily on the first call: CodSpeed's analysis runner invokes the
// bench function without tinybench's `setup`/`teardown` options.
bench(name, async () => {
wrapper ??= await mountSuspended(makeParent(render), { props: { cls: 'p-2' } })
await wrapper.setProps({ cls: 'p-3' })
await wrapper.setProps({ cls: 'p-2' })
}, {
async setup() {
wrapper = await mountSuspended(makeParent(render), { props: { cls: 'p-2' } })
},
teardown() {
wrapper?.unmount()
}
})
})
}
Expand Down
4 changes: 4 additions & 0 deletions vitest.config.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { fileURLToPath } from 'node:url'
import { defineVitestProject } from '@nuxt/test-utils/config'
import { defineConfig } from 'vitest/config'
import vue from '@vitejs/plugin-vue'
import codspeedPlugin from '@codspeed/vitest-plugin'
import ui from './src/vite'
import { glob } from 'tinyglobby'

Expand Down Expand Up @@ -51,6 +52,9 @@ export default defineConfig({
setupFiles: ['./test/utils/setup.ts']
},
plugins: [
// Instruments benchmarks when running under the CodSpeed runner in CI,
// inactive for a local `pnpm bench`.
codspeedPlugin(),
vue(),
ui({ dts: false }),
{
Expand Down
Loading