Monorepo first class support

[skimhugo]

Clear and concise description of the problem

make monorepo first class support

Suggested solution

make monorepo first class support

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[antfu]

Can you describe more about what you are expecting? Thanks

[JakeGinnivan]

FWIW, I have taken a crack at switching the NX plugins repo for my old company across. sevenwestmedia-labs/nx-plugins#61

My main issue so far are console output differences when running vitest via their command runners.

I will likely create a vitest plugin soon which will automate the conversion from jest -> vitest in a NX mono repo and provide the executors to run vitest tests for that project

[ocavue]

Jest has a projects config option, which allows me to run test cases from multiple packages within one monorepo.

For example, my project could have following file structure:

my-monorepo/
    packages/
        pkg-a/
            e2e-tests/
            unit-tests/
            src/
            package.json
            jest.unit.config.mjs
            jest.e2e.config.mjs
        pkg-b/
            unit-tests/
            src/
            package.json
            jest.unit.config.mjs
    package.json
    jest.config.mjs

// my-monorepo/jest.config.mjs
export default {
    projects: ["./packages/*/jest*.config.mjs"],
}

// my-monorepo/packages/pkg-a/jest.unit.config.mjs
export default {
    testMatch: "**/unit-tests/**/*.[jt]s?(x)",
    displayName: { name: "pkg-a:unit", color: "blackBright" },
    testEnvironment: "jsdom",  
}

// my-monorepo/packages/pkg-a/jest.e2e.config.mjs
export default {
    testMatch: "**/e2e-tests/**/*.[jt]s?(x)",
    displayName: { name: "pkg-a:e2e", color: "blackBright" },
    testEnvironment: "playwright",
}

// my-monorepo/packages/pkg-b/jest.config.mjs
export default {
    testMatch: "**/unit-tests/**/*.[jt]s?(x)",
    displayName: { name: "pkg-b", color: "magentaBright" },
    testEnvironment: "node",
}

I can run Jest CLI under my-monorepo and it can schedule all tests from the whole monorepo at once. I found it's quite useful for coverage reports. I can get one big coverage report from the whole monorepo codebase, instead of multiple reports for each small package that I need to combine manually later.

[cawa-93]

This could be very useful. I have a project with several independent packages. Each of them is in its own directory, has its own vite.config.js in which the root parameter is set. At the moment, I have to run several separate commands for each package:

vitest run -r packages/package1
vitest run -r packages/package2
vitest run -r packages/package3

But, if possible, it would be fantastic to be able to list all the roots and run all the tests for each root.

[jakeboone02]

Sorry if this is basically a +1, but I'm evaluating vitest for react-querybuilder and I have a great use case for this feature. If you look at the repo, you'll see there are no "test" scripts defined in any of the package.json files under the packages directory, only in the root. The "test" script in the root package.json just runs "jest", which reads the root jest.config.js and loops through the packages specified by the projects property.

As @ocavue said, it's great for producing an aggregate coverage report across the monorepo by simply running yarn test from the root.

I'm really liking vitest so far, but missing first-class monorepo support is probably a deal-breaker at this time (for my monorepos like react-querybuilder, anyway). Unless someone knows how to combine individual coverage reports after the fact...

[zfben]

I like this feature and need it too.

Jest is not good for ESM but good for monorepo. Vitest is good for ESM but not support monorepo.

[rogerleung0411]

Any progress in this issue?

[sheremet-va]

Not the priority right now, but something we want to do. Current priority is making Vitest more stable by closing "bug" issues 😄

Right now most of the people are either on a vacation or doing other OSS projects. And for me the priority is stabilizing and some easy to-do issues. But running one instance of VItest for different packages requires rewriting a lot of the current logic.

Also, it's possible to use Vitest with monorepos - the example is Vitest repo itself, we are running tests with -r flag, using pnpm monorepo features.

---

But if someone want to tackle this issue, you are welcome 😄

[xiaoxiangmoe]

Also, It would be great if vitest can aggregate unit test reporter and coverage reporter in monorepo config.

[jerrythomas]

Monorepo's using Turbo support executing tests for all packages in the repo. Maybe something similar is available with NX.

Scripts in root package.json
"test": "turbo run test:ci",
Script in each package.json
"test:ci": "vitest run",

Running pnpm test in the root folder executes all package tests.

Coverage reports generated for individual packages can be combined using the script merge-lcov.sh.

[Yankovsky]

I believe that jest projects option accumulates all the tests and then runs them as a single jest process. So instead of calling separate commands like jest project1 and jest project2, it can gather all the tests and manage resources better.

[connorjs]

Long post 😅

TL;DR

1. How do we expect/want the API/configuration to look?
2. I wrote an initial, hacky custom provider (that extends the C8 provider) as a proof-of-concept for aggregating coverage data across a monorepo.

Overview

Hello all 👋🏻 - I am new to Vitest and was also looking for “aggregation” support across sub-packages in a monorepo. In this latest project, I use Turborepo to manage the build orchestration, but other tools (Nx, Learn, etc) fall into the same category.

I prototyped a solution in my own repository (I will build a public GitHub repo to showcase it later), and wanted to share my thoughts with the community to drive this forward. I think the first step is for us to decide how we expect/want the API/configuration to look.

Disclaimer: I am quite new to Vitest. I have not yet tried out the Vitest UI (--ui), so my thoughts may evolve after using that.

Configuration

What should Vitest support with respect to monorepos?

1. projects approach (similar to Jest). Essentially, you have a root configuration that defines your “projects” and the tool itself (Vitest in this case) orchestrates the test run across each directory.

   Using Nx’s terminology, I view this as the “integrated repo” approach. The sub-packages can still have configuration (e.g. displayName), but CI executes at the root.

   (TBD on bike-shedding the term, projects to match Jest or workspaces to match npm/yarn)

2. Out-of-the-box aggregation/merging support. Essentially, enable a solution similar to istanbul-merge where you can generate a (coverage) report* from multiple sources. C8 supports this too with c8 report.

   Using Nx’s terminology again, I view this as the “package-based” approach. The sub-packages own the configuration (possibly with/via shared configuration), but the repo owner must orchestrate the coverage: They run vitest in each sub-package (using their tool of choice) and then run a final command at the root that aggregates/merges the individual reports together.

   *I say “(coverage) report” because that was my active needs (whether test output or coverage reporters). I still need to check out the Vitest UI to see how this option relates there.

Note: Maybe (2) goes first as foundation for (1).

My prototyping

I prototyped something similar to (2) yesterday and was quite happy with the results.

The key is to write the raw coverage data to files that can be processed on a final pass. At time of writing, Vitest actually overrides C8’s internal method to not write to disk because it has the coverage data in memory (which has better performance I assume).

The final Vitest solution may keep everything in memory too, but for my prototyping I wrote to disk to stay out of Vitest core (for now).

I can create a full repo later, but here’s the gist of my solution (warning: it has hacks).

Extend the C8 provider

I extended the C8 provider to write the coverage data to a project level directory for later processing.

I am using C8, so I focused on just extending C8. I assume this could be generalized across all providers with the existing Vitest abstractions in place.

// Using a subclass to illustrate the change
class C8CoverageMonorepoProvider extends C8CoverageProvider {
	// TBD on the specific monorepo options and how they would play with existing
	// vitest options (`root`, `all`, `include`, and `src` come to mind).
	readonly projectLevelDirectory: string;

	constructor(projectLevelTemporaryDirectory: string) {
		super();
		this.projectLevelDirectory = projectLevelTemporaryDirectory;
	}

	async onAfterSuiteRun(afterSuiteRunMeta: AfterSuiteRunMeta) {
		super.onAfterSuiteRun(afterSuiteRunMeta);
		if (!afterSuiteRunMeta.coverage) return;

		// The main change: Write the coverage data
		const coverageData = JSON.stringify(afterSuiteRunMeta.coverage);
		// Confirm the file name (likely, check C8 implementation)
		// For now, just hash it
		const hash = createHash(`sha256`, { encoding: `utf8` });
		hash.update(coverageData);
		const suffix = hash.digest(`hex`).slice(0, 10);
		// And write 😄
		await writeFile(
			join(this.projectLevelDirectory, `coverage-${suffix}.json`),
			coverageData,
		);
	}
}

The biggest hack here for those trying to reproduce was pulling the class out of the provider file. An integrated solution would not need to do this. (It would have proper support and not need sub-classing.)

// eslint-disable-next-line @typescript-eslint/ban-ts-comment -- Hacking
// @ts-ignore
import { C8CoverageProvider } from '@vitest/coverage-c8/dist/provider-a020853a.js';

Custom coverage provider

I wired this up as a custom coverage provider.

const C8CoverageMonorepoProviderModule: CoverageProviderModule = {
	async getProvider(): Promise<CoverageProvider> {
		// TBD how to configure the temporary directory at project root
		const directory = `${workspaceRoot}/build/tmp`;

		// Clean/create it
		if (existsSync(directory)) {
			await rm(directory, { recursive: true, force: true });
		}
		await mkdir(directory, { recursive: true });

		// Return the customized provider
		return new C8CoverageMonorepoProvider(
			directory,
		) as unknown as CoverageProvider; // Hacking
	},
	startCoverage,
	stopCoverage,
	takeCoverage,
};
export default C8CoverageMonorepoProviderModule;

Merge results

Finally, I wrote a report-monorepo-coverage.ts script that generated a new report. Again, this is C8-specific as part of my own prototype.

// Target directories
const rootCoverageDirectory = join(`build`, `coverage`);
const temporaryDirectory = join(`build`, `tmp`);

// Find all the workspaces that have coverage
const workspaces = getWorkspaces(); // TBD: Find official npm solution for this
const testedWorkspaces = workspaces.filter((workspace) =>
	existsSync(join(workspace, `build`, `coverage`)),
);

// Generate the report
await new Report({
	reporter: [`html-spa`],
	reportsDirectory: rootCoverageDirectory,
	tempDirectory: temporaryDirectory,
	all: true,
	// `include`, not `src`. We only want workspaces with tests for reporting.
	// Using `src` could show dependency files that are exercised. (Note: This
	// could represent an option, maybe on/related to `all`, but I prefer the
	// “only include packages with tests.”
	include: testedWorkspaces.map((workspace) => join(workspace, `src`)),
}).run();

// Clean up temporary directory
rmSync(temporaryDirectory, { recursive: true, force: true });

Orchestration

I use turbo repo, so it handles the orchestration. Essentially, run vitest run --coverage in each sub-package and then run ts-node scripts/report-monorepo-coverage.ts at the root.

Final result

I had an aggregated coverage report (using the html-spa reporter, my preference) in the root build directory in addition to per-package coverage reports in their own build directory.

Conclusion

I will be doing the following next, but happy to chat more on this topic.

1. Explore Vitest UI so I can evolve my thoughts on Vitest + monorepo if needed.
2. Publish my prototype in a new GitHub repo to show the solution more fully.

I hope this helps move the conversation along! Feedback definitely welcome!

[sheremet-va]

Can you elaborate on what features you want to be scoped to the project? We are thinking about several possible implementations, but they all have their set of drawbacks.

1. Allow jest-style "projects" support. For this, we will have to create a separate Vite server instance for each project, which might affect performance drastically. This will also require a rewrite for how tests are running to still run workers in a single pool (currently it is tied to the Vitest instance).
2. Use the approach we already provide, but expand on it to include other options. I am talking about poolMatchGlob and environmentMatchGlob. You can already set the environment or pool (workers/threads/browser) for a specific set of files, but you cannot customize other options like setupFiles. This will be quite limiting because we will run only one test server, so you cannot have separate plugins or aliases for different projects, which is quite limiting for a monorepo support.

[sheremet-va]

For this, we will have to create a separate Vite server instance for each project, which might affect performance drastically.

To bypass that, we can allow a special property for the projects config:

export default {
  projects: [
    ['packages/*', { config: './shared.js' }], // will override config,
    ['packages2/*', { extends: './shared.js' }], // will merge shared config with values defined in the packages2
    ['packages3/*'] // require config in the directory
  ]
}

This might allow reusing the same Vitest instance if the config is the same.

[Yankovsky]

From the discussion above I think the two most important features are:

• combined coverage reports
• better performance by running tests in one go

[sheremet-va]

So, support for monorepo is released in 0.30.0. All feedback is welcome. We did not implement any performance optimizations just yet, so I expect if you have a lot of projects in a monorepo you might get various degrees of performance issues because Vitest starts a new Vite server for each project, but I did not test this just yet. If you hit any issues with a monorepo setup, please open a separate issue.