[Unit Testing] Why not use vitest instead of jest

[ampmonteiro]

Which @angular/* package(s) are relevant/related to the feature request?

upgrade

Description

i read in blog post about two tings:

• Better unit testing with Jest and Web Test Runner
• Developer preview of the esbuild-based build system, with vite

so my question is: why not use a tool which has better integration with vite as well as better performance ?

Proposed solution

Replace jest by using vitest, which is:

• compatible with jest API
• better performance
• the best fit when using vite

Alternatives considered

allow the devs choose which want to by make available Jest and vitest.

Where those which:

• dont use vite build base, it is recommend use Jest

• Otherwise vite + vitest, would be the best bet.

[pumano]

would be good to have both options

[dgp1130]

Apologies for the delay, I was out on vacation and then sick for the last few weeks. I've talked about this a bit on social media, but it's good to put this in GitHub so I'll give a more complete answer here.

We did consider Vitest as a possible alternative to Jest. There's a lot to like about Vitest and it would probably make a viable unit testing solution for Angular. We ultimately decided on Jest for a few reasons.

Usage

Jest is used at significantly larger scale than basically any comparable unit test runner, including Vitest. Comparing the two, Jest has an order of magnitude more usage in the JavaScript ecosystem (currently 19.9k NPM downloads compared to Vitest's 1.7k downloads). While usage certainly isn't everything, it does demonstrate that Jest is successfully solving testing problems at significant scale and we can say that with a higher confidence than we can for Vitest.

One of Angular's core use cases is large scale, enterprise applications. We also already have significant usage in this space and any changes to Angular's tooling story need to not just meet the needs of 90% of our existing users, but 99%+. In choosing a new unit test runner, we need to be confident that we can meet that high bar, and seeing a tool already meeting those needs gives a lot more confidence than a comparable tool with less usage in that space.

The Angular community specifically has significantly more existing Jest usage. As of today, there are several community supported options for using Jest for an Angular applications, mostly based on jest-preset-angular. This demonstrates again that Jest is solving real problems for Angular applications at scale. In comparison, community support for Vitest in Angular is still quite minimal. The only one I'm aware of is Analog, which is still quite young and does not have nearly as much usage or validation in the community.

Community interest

The Angular community has overwhelmingly asked for Jest support. This has been a top user request for years. In our Angular Developer Survey, users frequently cited unit testing as a pain point and many explicitly requested Jest support as a solution. While Vitest and other test runners do occasionally come up in these conversations, our discussions with developers consistently favor Jest. I'd estimate Jest comes up at probably a 25:1 scale to alternatives in my personal conversations with the community.

A related point is that Jest seems to be much more familiar to the JavaScript community than Vitest. As one example, State of JS 2022 showed that 92.8% of respondents were familiar with Jest, while only 46.8% were aware of Vitest. This makes sense as Vitest is significantly younger than Jest. Given more time, I'm sure Vitest's usage and awareness in the community will grow, it is already trending very strongly. That said, it is hard to predict exactly how Vitest will compare to Jest long term, we can only make decisions based on the data we have available now.

Wait, didn't the CLI add Vite already?

I can definitely understand some confusion given that at the same time we announced experimental Jest support, we also started using Vite as the devserver, which can sound very contradictory. To explain some of the process here, we were trying to answer two major questions:

1. How do we handle devserving with the new ESBuild stack? Historically we used webpack-dev-server which is no longer viable without Webpack, so we need something else.
2. What improvements should we make to unit testing to meet user needs? If we decide to add a new unit testing solution, which should we choose?

While these two decisions were made at approximately the same time by the same individuals, they were very deliberately decoupled from one another to descope the complexity of each and ensure that we don't hold up either project by merging two smaller and fundamentally unrelated problems into one significantly larger solution. This enabled the team to answer both questions on a timeframe that allowed us to ship both a Vite-based devserver and experimental Jest support in v16. If we had coupled both questions together, it is very likely neither would have landed in v16.

It is also important to keep in mind that Vite is used in many different ways in the JavaScript ecosystem. Some applications use it as a complete build system, with custom plugins and integrations. The fact that it also contains a solid devserver and a test runner in Vitest is a great option when you're already invested in that stack. However, that is not the design of the Angular CLI. We already have a comprehensive build system and use an independent bundler (ESBuild). Vite is being used solely in its devserving capacity, not as a full build system, bundler, or plugin ecosystem. As a result, just because we are using Vite does not mean that Vitest is significantly easier to integrate or that it will perform as well. In fact it's actually undesriable from a maintenance perspective to couple Angular's devserver implementation with its test runner. Doing so only makes sense when Vite is used as a comprehensive build system, which just doesn't apply in this case.

Also on the performance front, Vitest is generally considered faster than Jest. I haven't done a full performance analysis here, but it's important to note that Vite's primary performance wins come from doing file-by-file transpilation and disabling TypeScript type checking. Due to the way Angular is designed, neither is actually possible right now. Even if you try to transpile a single TypeScript file, Angular must do a full program analysis (including type checking) before emitting even a single output file. Even if we did use Vite as a build system and did file-by-file transpilations with Vitest, users actually wouldn't see the most significant performance wins anyways. Again, this is partially why we are using Vite purely as a devserver. We are hoping to migrate Angular away from full program analysis (angular/angular#43165) and have made significant strides in that direction already, but it's important to note that as of today it is not clear that Vitest would perform significantly better than Jest.

Why not both?

As a few people have mentioned, why not give the users the option to choose between Jest and Vitest? This is always possible, choosing to support Jest now does not necessarily preclude supporting Vitest in the future. We do listen to the community, if there is sufficient desire for it and users believe it would solve real problems for them which Jest does not, then there's no fundamental reason we can't support both. That said, there are two key reasons for why we aren't committing to adding Vitest right now.

First, it is not clear to me how users would decide between Jest and Vitest and what the meaningful differences would be. If one is strictly better than the other, then why would we even give users a choice to begin with? Just always use the better one. If Angular supports multiple options, there must be a meaningful trade-off and decision users must make between them.

For example, we already support Karma and are hoping to migrate that implementation to Web Test Runner. WTR uses a real browser for executing tests, something we've had success with inside Google and which Angular has used historically. When presented with a choice between WTR and Jest, the real question is "Do you want to test with a real browser, or in an emulated Node environment?" There are lots of trade-offs and complexities to that question, and it is very difficult for the Angular team to say "Everyone should do X", so instead we present both and allow users to make the choice which makes sense for them.

By contrast, asking users to choose between Jest and Vitest is much less clear. Under what use case-specific conditions would users choose one unit testing solution over the other? Vitest is designed to emulate the Jest API and primarily differentiates itself by being easy to use for Vite-based projects and providing solid tooling and performance. Those are all concerns handled by the Angular CLI itself and its integration with the test runner, so even if we supported both, users wouldn't see meaningful benefits in one over the other. And again, if one happens to perform better and integrate more smoothly, we should just always use that option and not even bother with the other one.

The second reason is much simpler: maintenance effort. Ultimately building and maintaining a unit testing solution and its integration with Angular is a lot of work and we'd like to keep this down to the minimum amount. We already support browser-based testing in Karma and are planning to migrate this over to Web Test Runner. Adding Jest means we'll have at least 3 unit testing stacks in Angular CLI, eventually 2 once Angular browser testing support is completely migrated to WTR. Adding yet another unit testing stack is a significant amount of work we could be spending on other highly requested features, so we want to make sure we're investing our time and energy where it will have the most impact on the Angular community.

If we did add Vitest in the future, I expect it would probably be in a situation where we find that it is a better overall Node-based unit testing solution than Jest. In that situation, we would likely choose to deprecate Jest and migrate those users to Vitest entirely given the strong Jest compatibility already present in Vitest. That would minimize the number of supported testing stacks and remove the question of "Should I use Jest or Vitest?" from Angular users. To be clear, we currently have no plans to actually do that at this time, my point is that I see it as a more likely outcome than supporting both Jest and Vitest at the same time given their current similarities.

So it's Jest forever?

Of course not! On the Angular team, we're always adjusting our strategy over time, responding to community requests and the ever-evolving web tooling ecosystem. Right now we believe the best option is Jest, but things are always subject to change. Vitest is still quite young and has shown significant promise. In the next few years, I expect we'll see Vitest grow significantly and that may very well change the calculus of this decision. If it does, we'll revisit and make the appropriate changes. Whether that's migrating Jest users to Vitest, adding Vitest as a completely separate option, or maybe even using a completely different test runner! Ultimately we can only make decisions with the information we have right now and do our best to set up Angular for success in the future. Today we think that's Jest, but tomorrow we'll have to see.

[michaelfaith]

Really appreciate the thorough response. I had the same question myself about vitest since the bit that I've used has been a rewarding experience, and the synergy with vite seemed like a slam dunk. But knowing, now, that Vite is used in a more limited capacity helps clarify that. And, to your point, we've been using Jest in a significant portion of our Angular applications for years, and it's been a solid option for us (in no small part to the great work that @ahnpnl's done with jest-preset-angular). So it's definitely proven to be good at scale.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}