chore(github): add CodSpeed workflow - #6785
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
benjamincanac
changed the title
📝 WalkthroughWalkthroughThe change adds the CodSpeed Vitest plugin and configures Vite peer dependency support for Vite 7 and Vite 8. The Vue test project loads the plugin for benchmark instrumentation. Component benchmarks now mount wrappers lazily during the first benchmark call. A GitHub Actions job installs dependencies, prepares the Nuxt repository, and runs Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
benjamincanac
changed the title
benjamincanac
changed the title
commit: |
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/bench/button-link.bench.ts`:
- Around line 34-46: Separate initial mounting from re-render measurement in
reRenderBench and the corresponding sites in test/bench/components.bench.ts
lines 26-34 and 73-77 and test/bench/link-stack.bench.ts lines 48-53: add
dedicated mount benchmarks, ensure each re-render benchmark starts with an
already mounted component, and measure only the loading toggle cycle without
lazy mount logic. The anchor site is test/bench/button-link.bench.ts lines
34-46.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 8e29000d-7ab6-4d28-9610-4d548866ba9a
📒 Files selected for processing (3)
test/bench/button-link.bench.tstest/bench/components.bench.tstest/bench/link-stack.bench.ts
| function reRenderBench(name: string, comp: any, props: Record<string, any> = {}) { | ||
| describe(`re-render: ${name}`, () => { | ||
| let wrapper: Awaited<ReturnType<typeof mountSuspended>> | ||
| let wrapper: Awaited<ReturnType<typeof mountSuspended>> | undefined | ||
|
|
||
| // Mounted lazily on the first call: CodSpeed's analysis runner invokes the | ||
| // bench function without tinybench's `setup`/`teardown` options. | ||
| bench(name, async () => { | ||
| wrapper ??= await mountSuspended(comp, { props }) | ||
| await wrapper.setProps({ loading: true }) | ||
| await wrapper.setProps({ loading: false }) | ||
| }, { | ||
| async setup() { | ||
| wrapper = await mountSuspended(comp, { props }) | ||
| }, | ||
| teardown() { | ||
| wrapper?.unmount() | ||
| } | ||
| }) | ||
| }) | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
CodSpeed vitest-plugin walltime simulation mode warmup or setup invocation behavior
💡 Result:
When using CodSpeed with Vitest, the instrumentation and measurement modes are primarily handled by the CodSpeed CI environment and the @codspeed/vitest-plugin [1][2]. The invocation behavior for setup and warmup varies based on the measurement mode and the specific configuration [3][4]. 1. Warmup and Setup Invocation CodSpeed manages the lifecycle of your benchmarks to ensure measurement consistency [5][3]. While you define your benchmarks using standard Vitest bench syntax, CodSpeed handles the warmup and repetition cycles automatically in the CI environment [5][3]. For complex setups (such as database connections or external services), you can use Vitest's setupFiles configuration or standard lifecycle hooks like beforeAll, optionally combined with the setupInstruments utility provided by the @codspeed/vitest-plugin if specific instrumentation initialization is required [6]. 2. Simulation vs. Walltime Modes The invocation behavior and underlying measurement approach differ significantly between modes: - CPU Simulation (Default): This mode measures performance by simulating CPU cycles [3][7]. It is hardware-agnostic, deterministic, and typically produces results with <1% variance [3][7]. It does not measure system calls or I/O-bound operations and is best suited for CPU-intensive tasks [5][3][7]. - Walltime Mode: This mode measures the actual "wall clock" elapsed time, capturing system calls, I/O, and multi-threading effects [5][3][4]. Because it is sensitive to environmental noise, it requires consistent hardware [5][4]. CodSpeed recommends using their managed "codspeed-macro" runners when executing in this mode to ensure high precision [8][5][4]. 3. Invocation in CI To enable these modes, you specify the desired mode in the CodSpeed GitHub Action [9][8]. When running in CI, the CodSpeed plugin detects the instrumented environment and replaces or enhances the default Vitest runner [1][2][10]. If you run the command locally or outside of a supported CI environment, the plugin falls back to the default Vitest runner, and CodSpeed-specific instrumentation is not applied [1][2][10]. In summary, you do not manually invoke warmup or simulation logic; instead, you configure the mode via the CodSpeed GitHub Action and rely on the @codspeed/vitest-plugin to manage the execution lifecycle during your CI runs [9][8][3]. If you have tasks that require specific preparation, standard Vitest lifecycle mechanisms are the appropriate way to handle them [6].
Citations:
- 1: https://codspeed.io/docs/benchmarks/nodejs/vitest
- 2: https://registry.npmjs.org/%40codspeed%2Fvitest-plugin
- 3: https://github.com/CodSpeedHQ/codspeed/blob/main/skills/codspeed-setup-harness/SKILL.md
- 4: https://codspeed.io/docs/instruments/walltime
- 5: https://github.com/codspeedhq/codspeed
- 6: https://codspeed.io/docs/instruments/databases/mongodb/nodejs/vitest
- 7: https://codspeed.io/docs/instruments/index
- 8: https://codspeed.mintlify.dev/docs/instruments/walltime
- 9: https://github.com/marketplace/actions/codspeed-performance-analysis
- 10: https://registry.npmjs.org/@codspeed/vitest-plugin
🌐 Web query:
CodSpeed vitest-plugin analysis runner setup teardown bench invocation order
💡 Result:
The CodSpeed vitest-plugin operates by instrumenting your benchmarks to capture performance metrics accurately, specifically isolating the measured code execution from framework overhead [1]. Regarding the invocation order and lifecycle, the plugin is designed to bracket the measurement window specifically around the benchmarked loop [1]. This ensures that setup and teardown hooks—whether provided by the Vitest bench environment or user-defined logic—are typically excluded from the recorded walltime measurements [1]. Key points regarding the runner, setup, and teardown behavior include: 1. Instrumentation Scope: The CodSpeed runner wraps the measured function within a root frame, specifically ensuring that internal setup/teardown hooks (and post-loop statistical computations performed by the underlying runner, such as tinybench) fall outside the measured sample window [1]. 2. Marker Protocol: The plugin employs a specific marker sequence—SampleStart > BenchmarkStart > BenchmarkEnd > SampleEnd—to ensure that instrumentation remains balanced even if a benchmark throws an error [2]. This is handled via try/finally blocks within the runner [2]. 3. Execution Environment: When run locally, the plugin acts as a no-op (falling back to the default Vitest runner) [3][4][5]. The CodSpeed instrumentation only activates in CI environments where performance data capture is required [3][5]. 4. Setup/Teardown Handling: Because CodSpeed integrates with Vitest's existing lifecycle, standard Vitest setup/teardown (like beforeEach/afterEach) is respected, but CodSpeed’s instrumentation is strategically placed to avoid including the timing of these hooks in the final benchmark metrics [1]. For users requiring fine-grained control over setup/teardown outside of standard Vitest hooks, CodSpeed typically recommends the pattern used in its other framework integrations (such as the pedantic mode in pytest-codspeed) [6][7], though within Vitest, you should rely on the plugin's automatic management of the measurement window to ensure accurate results [1]. If you are encountering issues with setup/teardown being called multiple times, ensure the plugin is applied only to your benchmarking project in your vitest.config.ts, rather than globally, to avoid duplicate teardown events across multiple test projects [3].
Citations:
- 1: CodSpeedHQ/codspeed-node@2475d5b
- 2: CodSpeedHQ/codspeed-node@b7f1be7
- 3: https://github.com/Aureliolo/synthorg/blob/main/web/vitest.config.ts
- 4: https://www.npmjs.com/package/@codspeed/vitest-plugin
- 5: https://codspeed.io/docs/benchmarks/nodejs/vitest
- 6: https://codspeed.io/docs/reference/pytest-codspeed
- 7: https://codspeed.io/docs/guides/how-to-benchmark-python-with-pytest
🌐 Web query:
CodSpeed vitest plugin source runner setup teardown call bench
💡 Result:
The @codspeed/vitest-plugin integrates directly with Vitest's existing benchmarking capabilities, meaning you do not need special CodSpeed-specific calls for setup and teardown [1][2]. Instead, you leverage standard Vitest hooks and benchmarking patterns, which CodSpeed then instruments automatically when run in a supported CI environment [1][2]. To set up CodSpeed with Vitest: 1. Installation: Install the plugin and Vitest (v3.2+) as dev dependencies [1][2][3]. npm install --save-dev @codspeed/vitest-plugin vitest 2. Configuration: Add the plugin to your vitest.config.ts (or.mts) file [2][4]: import { defineConfig } from "vitest/config"; import codspeedPlugin from "@codspeed/vitest-plugin"; export default defineConfig({ plugins: [codspeedPlugin], }); 3. Writing Benchmarks: Continue using the standard Vitest bench function and standard test hooks like beforeEach, afterEach, beforeAll, and afterAll [2][4]. CodSpeed intercepts these processes during CI execution to provide performance instrumentation [1][2]. Example structure: import { describe, bench, beforeAll, afterAll } from "vitest"; describe("my-benchmark", => { beforeAll(async => { // Standard Vitest setup }); afterAll(async => { // Standard Vitest teardown }); bench("my-task", => { // Perform task }); }); When running locally, CodSpeed detects the environment and gracefully falls back to the default Vitest runner, outputting standard benchmark results [1][2][5]. CodSpeed instrumentation is specifically intended for CI pipelines to ensure consistent, reliable performance measurements [2][3].
Citations:
- 1: https://www.npmjs.com/package/@codspeed/vitest-plugin
- 2: https://codspeed.io/docs/benchmarks/nodejs/vitest
- 3: https://registry.npmjs.org/@codspeed/vitest-plugin
- 4: https://codspeed.io/docs/instruments/databases/mongodb/nodejs/vitest
- 5: https://cdn.jsdelivr.net/npm/@codspeed/vitest-plugin@5.3.0/README.md
🌐 Web query:
CodSpeed walltime runner bench callback invoked multiple times simulation mode
💡 Result:
The behavior you are observing is a fundamental difference between CodSpeed's two primary measurement instruments: Simulation and Walltime [1]. In CodSpeed, benchmarks are invoked differently depending on the measurement mode [1][2]: 1. Simulation Mode (Default): This instrument simulates CPU behavior to provide consistent, hardware-agnostic measurements [1][3]. In this mode, each benchmark is designed to run only once [3][4][5]. Because it measures simulated CPU cycles and cache/memory access patterns rather than real-time duration, multiple invocations are unnecessary and are typically avoided by the runner [1][3]. 2. Walltime Mode: This instrument measures the actual "wall-clock" time of your benchmark [6][7]. To ensure statistical significance and handle real-world noise (such as I/O or background processes), the CodSpeed runner intentionally invokes the benchmark callback multiple times [6][2]. The process typically involves [2]: - A Warmup Phase: The benchmark runs repeatedly until a steady state is reached [2]. - A Measurement Phase: Based on the warmup results, the runner calculates the number of rounds required to satisfy configured time or round constraints, executing the benchmark multiple times to collect timing data for each round [2]. If you see your benchmark callback being invoked multiple times, it is confirmed that you are running in (or have configured) Walltime mode [2]. If you require your benchmark to run only once, you must ensure you are using the Simulation instrument [3][4]. You can control this via: - The mode parameter in the CodSpeed GitHub Action (e.g., mode: "simulation") [8]. - The CODSPEED_RUNNER_MODE environment variable [9][10]. - CLI flags like --measurement-mode simulation [11][5] or C++ CMake flags [4]. Note that parallel execution of multiple benchmark processes is discouraged in Walltime mode as it introduces noise and reduces measurement precision [6][7].
Citations:
- 1: https://codspeed.io/docs/instruments/index
- 2: https://codspeed.io/docs/cli
- 3: https://codspeed.mintlify.dev/docs/instruments/cpu
- 4: https://codspeed.io/docs/benchmarks/cpp
- 5: https://codspeed.io/docs/benchmarks/rust/bencher
- 6: https://codspeed.io/docs/instruments/walltime
- 7: https://codspeed.mintlify.dev/docs/instruments/walltime
- 8: https://github.com/CodSpeedHQ/action
- 9: Different results running benchmarks with and without the action CodSpeedHQ/action#127
- 10: feat(setup): add optional
modearg to target setup CodSpeedHQ/codspeed#397 - 11: https://codspeed.mintlify.dev/docs/benchmarks/rust/criterion
🌐 Web query:
CodSpeed vitest plugin runner.js mountSuspended bench invocation CodSpeed
💡 Result:
The CodSpeed Vitest plugin and the mountSuspended utility are two distinct tools used in different contexts within the Vitest ecosystem. The @codspeed/vitest-plugin is used to instrument and run performance benchmarks [1][2]. To use it, you configure it as a plugin in your vitest.config.ts file [1][2]: import { defineConfig } from "vitest/config"; import codspeedPlugin from "@codspeed/vitest-plugin"; export default defineConfig({ plugins: [codspeedPlugin], }); You then run your benchmarks using the standard vitest bench command [1][2]. Locally, CodSpeed will fall back to the default Vitest runner, while in CI environments, it provides instrumented performance measurements [1][3]. The mountSuspended utility, on the other hand, is a specific tool from @nuxt/test-utils used to mount Vue components within a Nuxt environment during unit tests [4][5]. It allows for async setup and access to Nuxt context (such as plugins and auto-imports) [4][5]. It is not a benchmarking tool and is not typically used inside a CodSpeed bench invocation [4][6]. There is no direct integration between mountSuspended and the CodSpeed runner. If you intend to benchmark a component that requires mountSuspended (e.g., measuring the performance of a Nuxt component setup), you would likely need to manually manage the Nuxt environment within the bench task, although this is generally not the recommended pattern for performance benchmarking, which focuses on stable, isolated logic execution [2][5]. Performance benchmarks in CodSpeed are designed to measure code execution time, whereas mountSuspended is designed for functional testing within a specific framework lifecycle [2][5].
Citations:
- 1: https://www.npmjs.com/package/@codspeed/vitest-plugin
- 2: https://codspeed.io/docs/benchmarks/nodejs/vitest
- 3: https://registry.npmjs.org/%40codspeed%2Fvitest-plugin
- 4: https://github.com/nuxt/test-utils/blob/main/src/runtime-utils/mount.ts
- 5: https://deepwiki.com/nuxt/test-utils/3.1-mounting-components
- 6: https://michaelnthiessen.com/tips/nuxt-mount-suspended
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package files =="
git ls-files | rg '(^package\.json$|pnpm-lock\.yaml$|package-lock\.json$|yarn\.lock$)' | sed -n '1,50p'
echo
echo "== CodSpeed/tinybench references =="
rg -n "`@codspeed`|codspeed|tinybench|CODSPEED|measurement-mode|mode:" -S . \
--glob '!node_modules/**' --glob '!dist/**' --glob '!build/**' | sed -n '1,120p'
echo
echo "== relevant bench files snippets =="
for f in test/bench/button-link.bench.ts test/bench/components.bench.ts test/bench/link-stack.bench.ts; do
if [ -f "$f" ]; then
echo "--- $f ---"
sed -n '1,110p' "$f" | cat -n
fi
done
echo
echo "== vitest config and script snippets =="
if [ -f vitest.config.ts ]; then sed -n '1,200p' vitest.config.ts | cat -n; fi
if [ -f package.json ]; then jq '.scripts, .devDependencies, .dependencies' package.json | sed -n '1,220p'; fiRepository: nuxt/ui
Length of output: 25170
🌐 Web query:
CodSpeed vitest plugin 5.7.1 benchmarks runner invocations bench callback setup teardown walltime simulation
💡 Result:
The CodSpeed vitest plugin facilitates benchmarking by integrating with the Vitest test runner. It requires Vitest v3.2 or higher [1][2]. While it allows developers to write benchmarks using the standard vitest bench function [1][3], the plugin specifically handles the instrumentation required to export these results to the CodSpeed platform for performance analysis in CI/CD environments [1][2]. Regarding runner invocations and performance measurement: Runner Invocations and Hooks: The CodSpeed vitest plugin maintains compatibility with Vitest's benchmark hooks, such as setup and teardown, while ensuring that performance metrics are captured accurately [4]. Recent optimizations have been implemented to ensure that the walltime measurement window is strictly bracketed around the measured loop itself [4]. This prevents framework overhead—such as the execution of setup/teardown hooks or post-loop result computation—from polluting the collected metrics, which previously resulted in inaccurate flamegraphs [4]. Instruments: CodSpeed supports two primary measurement instruments [5]: 1. CPU Simulation: This is the recommended default for most benchmarks [5]. It simulates CPU behavior to provide consistent, hardware-agnostic results regardless of system load [6][5]. Benchmarks using this mode typically run once, making it unsuitable for highly I/O-intensive code where simulation overhead may be significant [5]. 2. Walltime: This instrument measures real-world elapsed time [6][7]. It is best suited for benchmarks that involve significant I/O, network calls, or multi-threading, as it captures the actual performance observed in a real environment [8][5]. When using the walltime instrument, CodSpeed typically recommends using their bare-metal "Macro Runners" to ensure high precision and low noise in the measurements [7]. Locally, when running vitest bench without the CodSpeed CI instrumentation, the plugin gracefully falls back to the default Vitest runner, allowing developers to verify their benchmarks without requiring the full CodSpeed profiling infrastructure [1][2].
Citations:
- 1: https://codspeed.io/docs/benchmarks/nodejs/vitest
- 2: https://registry.npmjs.org/@codspeed/vitest-plugin
- 3: https://www.npmjs.com/package/@codspeed/vitest-plugin
- 4: CodSpeedHQ/codspeed-node@2475d5b
- 5: https://codspeed.io/docs/instruments/index
- 6: https://codspeed.mintlify.app/cli
- 7: https://codspeed.io/docs/instruments/walltime
- 8: https://github.com/codspeedhq/codspeed
🌐 Web query:
CodSpeed github runner walltime simulation bench invocations setup teardown tinybench
💡 Result:
CodSpeed provides two primary modes of measurement for performance benchmarking in CI: CPU simulation (default/recommended) and walltime [1][2]. Walltime Measurement and Macro Runners Walltime measurement captures the actual elapsed (wall-clock) time, including I/O operations, system calls, and multi-threaded execution effects [3][4]. Because traditional shared cloud runners introduce significant performance noise that can distort walltime data, CodSpeed requires the use of CodSpeed Macro Runners for this mode [5][6]. These are dedicated, 16-core ARM64 bare-metal machines managed by CodSpeed that provide a stable, isolated environment [5]. To enable walltime in a GitHub Actions workflow, you must: 1. Change the runner to use the CodSpeed-managed environment by setting runs-on: codspeed-macro [7][5]. 2. Configure the CodSpeed Action to use the walltime instrument by setting mode: walltime [7][1]. Tinybench Integration CodSpeed integrates with tinybench via the @codspeed/tinybench-plugin [8][9]. This plugin allows you to wrap your tinybench suite, enabling CodSpeed instrumentation [8]. Key aspects of the integration include: - Setup/Teardown Handling: The plugin is designed to exclude framework overhead from the measurements. It specifically targets the measured loop, ensuring that run-mode setup and teardown hooks (as well as statistical computation post-loop) are bracketed outside the walltime instrumentation window [10]. - Instrumentation Window: Recent updates to the plugin ensure that benchmark markers are emitted inside the sample window to maintain proper nesting (SampleStart > BenchmarkStart > BenchmarkEnd > SampleEnd), which is critical for accurate reporting [11]. - Versioning: The plugin supports multiple versions of tinybench (v4+), with internal adjustments to handle changes in tinybench's private field layouts and option accessibility in newer versions [8][12]. - Local Fallback: When run outside of the CI environment or without the full CodSpeed CLI/action instrumentation, the plugin automatically falls back to standard tinybench execution [8][9]. For optimal consistency, it is recommended to ensure your cache keys include ${{ runner.arch }} because Macro Runners use the ARM64 architecture [7][5].
Citations:
- 1: https://github.com/CodSpeedHQ/action
- 2: https://github.com/marketplace/actions/codspeed-performance-analysis
- 3: https://codspeed.mintlify.dev/docs/instruments/walltime
- 4: https://github.com/CodSpeedHQ/codspeed
- 5: https://codspeed.io/docs/features/macro-runners
- 6: https://codspeed.io/changelog/2024-11-27-walltime-instrument-and-codspeed-macro-runners
- 7: https://codspeed.io/docs/instruments/walltime
- 8: https://codspeed.io/docs/benchmarks/nodejs/tinybench
- 9: https://registry.npmjs.org/%40codspeed%2Ftinybench-plugin
- 10: CodSpeedHQ/codspeed-node@2475d5b
- 11: CodSpeedHQ/codspeed-node@b7f1be7
- 12: CodSpeedHQ/codspeed-node@d0a91d3
Do not rely on a single-run CodSpeed invocation for lazy mounts. CodSpeed simulation/bench runners invoke the benchmark function multiple times per sample loop, so wrapper ??= await mountSuspended(...) can include the one-time mountSuspended cost in measured re-render timings. Keep component re-render benchmarks benching the toggle cycle over an already mounted component, and run mountSuspended separately as its own benchmark.
📍 Affects 3 files
test/bench/button-link.bench.ts#L34-L46(this comment)test/bench/components.bench.ts#L26-L34test/bench/components.bench.ts#L73-L77test/bench/link-stack.bench.ts#L48-L53
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/bench/button-link.bench.ts` around lines 34 - 46, Separate initial
mounting from re-render measurement in reRenderBench and the corresponding sites
in test/bench/components.bench.ts lines 26-34 and 73-77 and
test/bench/link-stack.bench.ts lines 48-53: add dedicated mount benchmarks,
ensure each re-render benchmark starts with an already mounted component, and
measure only the loading toggle cycle without lazy mount logic. The anchor site
is test/bench/button-link.bench.ts lines 34-46.
1 participant
🔗 Linked issue
Relates to #6293
❓ Type of change
📚 Description
Runs the existing
test/benchsuite (tv slots, Button/Table mount and re-render, link stack) on CodSpeed so perf changes get a before/after report in CI. Local runs are too noisy to compare: theplain <button>control bench varies almost 2x between two consecutive runs on the same machine, while CodSpeed's simulation mode counts instructions deterministically.Same shape as the
test-benchmarkjob in nuxt/nuxt's ci.yml: abenchmarksjob inmodule.ymlrunningpnpm run benchthroughCodSpeedHQ/actionin simulation mode, with the baseline coming from the push runs onv4. Auth goes through OIDC instead of aCODSPEED_TOKENsecret.@codspeed/vitest-pluginonly activates under the CodSpeed runner, a localpnpm benchis unchanged.One catch: the plugin caps its
vitepeer at ^7 while the repo resolves vite 8 for the Nuxt modules. Without intervention, pnpm's peer dedupe re-keys the whole vite peer group down to vitest's vite 7, re-binding@nuxt/kit,@tailwindcss/viteand friends across 1,600 lines of lockfile churn. A scoped override widens the plugin's peer range to^7 || ^8so everything stays keyed on vite 8. The plugin runs inside vitest's own vite anyway, so the range is bookkeeping only.Replaces the auto-generated #6732, which added a parallel root-level
bench/setup instead of reusing the vitestvueproject. Once this lands onv4, PRs updated past it (like #6737) get a measured report on their hot path.📝 Checklist