Re-enable continuous benchmarking

[adriencaccia]

Describe the feature

Benchmarks have been added in #24846. It came with CodSpeed and a CI job that ran on every pull request to track performance.

However, it was disabled in a39d311. The reason is that there were some performance regressions falsely reported in unrelated PRs due to the flakiness of some benchmarks.
The flaky benchmarks are those defined in https://github.com/nuxt/nuxt/blob/main/packages/kit/test/load-nuxt-config.bench.ts. The flakiness comes from the fact that they rely on file system access, which at the moment is not reliably supported in CodSpeed. You can see an example of the flakiness here: https://codspeed.io/nuxt/nuxt/benchmarks/packages/kit/test/load-nuxt-config.bench.ts::loadNuxtConfig::basic%20test%20fixture%20(types)

To re-enable the CI job, one of the following options could be done:

• Remove those benchmarks entirely
• Change those benchmarks, and the underlying tested functions, to not rely on File System access

I think the second option would be best, but it will require some changes to the https://github.com/nuxt/nuxt/blob/main/packages/kit/src/loader/config.ts file.

In the future, we plan on adding better support on CodSpeed for flaky benchmarks due to FS access, but that may take some time before it's available.

Additional information

• Would you be willing to help implement this feature?
• Could this feature be implemented as a module?

Final checks

• Read the contribution guide.
• Check existing discussions and issues.

[adriencaccia]

Hey @danielroe,

We just released a feature to ignore a benchmark on CodSpeed. An ignored benchmark is removed is still run, but its measure is removed from the performance report.

This feature should allow you to re-enable CodSpeed, ignore the flaky packages/kit/test/load-nuxt-config.bench.ts::loadNuxtConfig::basic test fixture (types) benchmark and start benefitting from CodSpeed again.
We will also be able to collect more data on the specific flaky benchmarks, which will help us find the source of the problem and fix it 🤞

Let me know if you have any questions 😉