Benchmark Jest vs Vitest: Evaluate Test Runner Performance and Migration Feasibility

[dnim]

Summary

Evaluate whether switching from Jest to Vitest would provide meaningful speed, DX, or CI advantages in this repository.

---

Context

• Current test setup: Jest (ts-jest) for unit tests, Playwright for visual tests, run via npm scripts.
• Recent changes (PR chore: Configure Renovate #50): Renovate config, dependency pinning, no changes to test approach.
• Project type: Node.js CLI (not a Vite/web app). Jest is common here; Vitest is compelling mainly for Vite/modern ESM/TS projects or exceptional startup/watch speed.

---

Tasks

• Benchmark current Jest test runs (local, watch, hooks, CI)
• Add Vitest to a vitest-spike branch, port a representative unit test
• Run and compare:
  • Full suite cold and warm runs
  • Single test-file runs
  • Watch-mode feedback after edits
  • Pre-commit hook (if applicable)
  • CI workflow/test job durations
  • Config and mocking compatibility
• Document results: record medians (not single runs), config complexity, and migration risks/benefits
• Recommend keeping Jest or migrating to Vitest based on results

---

How to Benchmark (exclude benchmarking or migration work for current branch, this must be tracked as a separate issue)

1. In a vitest-spike branch, keep Jest alongside Vitest

2. Add scripts:

   {
     "test:unit": "jest --runInBand",
     "test:unit:vitest": "vitest run",
     "test:unit:vitest:watch": "vitest"
   }

3. Port 1–2 typical unit tests to Vitest

4. For both runners:

   • Measure time npm run test:unit
   • hyperfine --warmup 2 ... if available, or manual 3–5 runs and record averages
   • Try watch mode: note time from save to re-test
   • (Optional) Duplicate CI test job for A/B compare

5. Fill out this results table and add any migration issues:

   Scenario	Jest	Vitest	Notes
   Full suite cold run
   Full suite warm run
   Single test file
   Watch rerun after edit
   Hook/staged test run
   CI test job
   Config complexity
   Mock/timer compatibility

---

Decision Criteria

• Switch if: Vitest provides ≥20–30% faster runs, better watch-mode, or simpler config, and does not break or complicate test maintenance.
• Stay with Jest if: Gains are minor, migration cost/compatibility is high, or total pipeline isn’t limited by test speed.

---

References

• Current discussion and exploration
• PR chore: Configure Renovate #50 (Renovate) context: chore: Configure Renovate #50

---

This issue is based on an in-depth review of current CI/test config, goals, and peer repo/test runner tradeoffs.

Note: All benchmarking, migration, and analysis work related to new branches should be tracked in a separate issue for clarity and parallel progress. This issue is for global planning, not implementation steps tied to any single branch.