TestBed.configureTestingModule Performance Issue

[ollwenjones]

I'm submitting a ... (check one with "x")

[ ] bug report 
[x] feature request
[ ] support request 

Current behavior

Importing modules into TestBed.configureTestingModule can slow tests down substantially. We have a 'shared module' with parts, that lots of our components use, including a third party library or two. It is very convenient to import this module as a rather than cherry pick in the tests, however testing a small component yesterday, I saw test bootstrap taking two whole seconds and had to cherry-pick dependencies to make tests run reliably.

Expected behavior

TestBed.configureTestingModule should be performant.

Minimal reproduction of the problem with instructions

In this plunker

1. go to the src/simple.spec.ts and comment and un-comment the SharedModule import into configureTestingModule.
2. Observe time it takes for test to run.

I am seeing a jump from 0.079s to 0.241s. (~= 3x slower). Multiply that by 5 cases and this simple test takes a whole second. Make the shared module larger and you have real problems.

What is the motivation / use case for changing the behavior?

1. Slow tests hurt TDD workflow
2. Slow enough tests disconnect karma from the browser and fail the suite.

Please tell us about your environment:

Win10, VsCode, Webpack,

• Angular version: 2.0.X

  Yes.

• Browser: Locally, PhantomJS. Plunk tested in Chrome.

• Language: TypeScript (though tests are transpiled to ES5 prior to running, naturally)

Note: Even as I write this, I wonder if it's a reasonable expectation for large modules to go through TestBed quickly? Maybe I'm just doing something wrong architecturally, (use multiple, smaller shared modules, not one big one, etc.) However, tests are time-consuming enough without having to track down every individual dependency of a more complex component just to get started.

[ollwenjones]

Seems like the goal is really to have a fresh component instance each time, and for the most part the testing module is going to be static for a particular .spec file. It also seems like configuring the module at each iteration is the part that is really adding on the time. A better pattern could be configuring the module once, then testModule.createComponent multiple times.

TestBed.configureTestingModule once and calling TestBed.createComponent on that module multiple times doesn't work, though... maybe I'm missing an obvious way to preserve module state between iterations?

Edit: FWIW I tried logging the timestamp and it looks like most of the time passes during TestBed.createComponent, not TestBed.configureTestingModule - however I still suspect it's related to module configuration, since adding things to imported modules adds to the time, whether or not those imports are used in the component being tested.

[vmandy]

I have a noticed unit test execution performance issues that seem directly related to this issue.

I have an ionic 2 / angular 2 app. When trying to unit test my components using TestBed and ComponentFixture classes, there is easily 1-2 seconds spent in the setup of each test that seems directly related to the imports: [IonicModule.forRoot(MyComponent)] that I've added to my testing module via the TestBed.configureTestingModule({...}) method.

When I remove the import, my tests run much faster, but they obviously fail because my component's template no longer compiles.

[ollwenjones]

@vmandy I've been working around this by doing one or more of:

1. spending more time curating just what I need in the test module
2. mocking the component dependencies that are time-consuming to bootstrap
3. overriding the template of the component I'm testing so it doesn't depend on much.

All of which are time consuming and sub-optimal 👎 (except from the point of view that I am really forced to isolate the unit I am testing?)

It seems like there are some theoretical optimizations to this at least, because I know these components are not taking 1-2 secs to bootstrap at run-time.

[FlemmingBehrend]

Iam having similar problems. Thought it was caused by wallaby, but it seems to be the TestBed.

wallabyjs/public#885

[mlakmal]

I am seeing the same issue. Any unit test with component compilation is taking close to 1sec...

[ollwenjones]

Unfortunately I was pulled off the Angular 2 project and haven't looked at this in quite some time. It seems like a lot of the complications of TestBed (test-module setup, generally, not just performance) could be avoided by an approach similar to the enzyme.shallow render approach in the React ecosystem. - where just one unit is rendered and none of it's dependencies - the same thing can be accomplished by over-riding an Angular component's template in the .spec, but that is often tedious/time-consuming as well.

[mlakmal]

we have close to 2000 test cases getting executed just under 1 min in Angular 1 where Angular 2 takes 1min to execute just 100 test cases. It would be nice if we can get a resolution for this issue as it hinders the testing capabilities in large applications.

[ollwenjones]

@mlakmal I think there are lots of folks relying heavily on e2e tests, or just testing functional logic in component classes

let component = new ComponentClass(mockDep1, mockDep2);
expect(component.doAddition(2,2)).toBe(4);

Which doesn't test any of the template rendering (template logic)... but it does allow easy testing of class logic at least

[mlakmal]

@ollwenjones thanks, i am going to try that out. i really doesn't want to test any template logic since most of that gets covered by our automated tests, so just testing the component class code should be enough.

[ollwenjones]

@mlakmal sweet! Makes me happy to contribute something that actually helps someone. 🍻 - FWIW I'm often torn about template-rendering unit tests, because it's so easy to slip into testing the framework.

[michaelbromley]

I'm in the process of upgrading a pretty big project from rc.4 to 2.3, so it's the first time we're using the NgModule approach.

We have ~1000 unit tests, with maybe 1/2 of those testing components. Prior to upgrade, tests run pretty quickly - around 30s.

After upgrade, they take at leat twice as long, but the worst thing is that they seem kind of "flaky" - Karma will periodically lost the connection to Chrome and often I will need to refresh Chrome multiple times. I suspect that something is timing out but haven't isolated it yet.

@ollwenjones regarding "shallow" component tests, did you know about the NO_ERRORS_SCHEMA that you can set in the testing module?

import { NO_ERRORS_SCHEMA} from '@angular/core';
// ...
TestBed.configureTestingModule({
  declarations: [ /*... whatever */ ],
  schemas: [NO_ERRORS_SCHEMA]
});

This means that the compiler just ignores any elements it does not recognize, meaning you do not need to declare all the components used in the template of the component under test.

[marclaval]

Found the same issue in Angular's tests: #13500
I was able to improve the situation by creating bespoke testing modules for each test.

[ollwenjones]

@michaelbromley 😮 I had no idea about NO_ERRORS_SCHEMA would have / will probably save me hours - also an opportunity to hopefully speed up some of these tests, as the leaner the test-module the faster TestBed seems to go.

[FlemmingBehrend]

@michaelbromley that also helped me out a lot. Removing MaterialModule.forRoot() from my imports really speed up my tests. 👍

[awerlang]

Ideally, we should be able to call TestBed.configureTestingModule() inside a beforeAll(). It just doesn't work because of this line. This is reseting the modules before each test. Perhaps it shouldn't matter, but it does because Angular spends quite a lot of time compiling components in JiT mode. Reseting it throws away this information. Recording a profiling session on karma window led me to this conclusion.

IMO, this should be done:

1. Be able to configure a module in a beforeAll();
2. Cache compiled components

Another way to fix that is making reset testing module explicit, but this may break user code.

[gonzofish]

Is it clear that this isn't something to do with karma-phantom? The tests fly in Chrome for me, but Phantom takes at least a second per test. I've mocked the dependencies out as much as possible and they still run slow in Phantom.

[Necroskillz]

phantom is very slow for me (using karma-webpack, but this is a different issue. I use Chrome now, and you can visually see as the tests run when there are tests that use a lot of components that have to be compiled it gets stuck for a second and runs slowly until it hits service tests which run fast.

Configuring module in beforeAll is not so convenient, because i want to create new mocks after each run and have them (and other providers) in a fresh state.

Caching the compiled components is the way to go imo.

[philipooo]

You can also use `electron` and `karma-electron` to run your tests headless. Its nearly as fast as chrome. Phantomjs is about 8 times slower for me. Cheers Necros <notifications@github.com> schrieb am Mi., 8. März 2017, 23:49:

…

phantom is very slow for me (using karma-webpack, but this is a different issue. I use Chrome now, and you can visually see as the tests run when there are tests that use a lot of components that have to be compiled it gets stuck for a second and runs slowly until it hits service tests which run fast. Configuring module in beforeAll is not so convenient, because i want to create new mocks after each run and have them (and other providers) in a fresh state. Caching the compiled components is the way to go imo. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#12409 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABn5_18VjaBSPZQX1ZQ45mveN2PFMoNBks5rjzB-gaJpZM4KcS5I> .

[michalmo]

AOT support would help here too.

[thymikee]

@philipooo with slight modifications to some of Jasmine references (jasmine.createSpyObj() specifically), you can also run your tests without any browser – by using Jest instead of Karma.

For as few as 35 files with total 100 tests Jest has been able to speedup my tests 2.5x (vs Karma on Chrome). Tests are now running in parallel, completely isolated, not to mention superior watch mode with instant feedback.

I've written about it here, if anybody's curious: https://www.xfive.co/blog/testing-angular-faster-jest/.

There's also an issue to integrate it directly into CLI (e.g. under a flag: ng test --jest): angular/angular-cli#4543.

[ollwenjones]

@thymikee I'm super excited to hear that. I've had muuuch better experience testing React code with Jest than I ever have with Angular, and I've been persuaded that end-users of trustworthy frameworks don't need to do as much browser-testing.

Last time I checked Jest was using jsdom which did not play nice with zone.js is this no longer the case? 🤞 I'll have to read your blog.

[thymikee]

So jsdom is not the problem. It's just that tests must run in Zone.js context, which is done by this little lib: jest-zone-patch

[jsgoupil]

Testing with HTML components is just a pain... I have about 500 unit tests that are 90% looking at the DOM, it takes me 40 minutes to run them all... at this rate, Chrome crashes when it reaches a RAM level too high... the result is that I have to batch my tests by 50. Painful as hell...

There are definitely some memory leaks here and there...

[yairgo]

bueller?

[oferh]

Found out that using FormsModule considerably slows down compileComponents mainly using ngModel.

For example here is a plunker which has a component with 10 inputs and it takes about 3 times more when using ngModel.

[dblVs]

@juliemr Hey, I wanted to do a feature request to add a schema that does shallow rendering. Shall keep the stuff here (since what @ollwenjones wrote pretty much is what i want) or create a new issue?
Basically it's:

• Being able to instantiate the child components, BUT not it's template. (this is a bit more complex than it sound)
• Still having errors when bound to wrong property on the component (unlike error schema)

At the moment I'm creating stub components in my project to create blank components and have them implement same interface to make them consistent the inputs and outputs and makes it easier to my users to test the template of their component. It becomes really difficult once you include stuff like ContentChildren logic combined with templateOutlet directive usage or transclusion.

Possible solution is use normal schema ot 1st component then for the child components use the NO_ERROR_SCHEMA.

[ersimont]

I've put together a little helper to re-use compilation results for given modules for all your tests. I'd love to hear if it helps others. It's the precompileForTests() function in s-ng-dev-utils.

It seems like the APIs available for things like this keep changing, so we'll see how long this can last. I created this with Angular 8. It sounds like better support for AoT in tests may be coming with Ivy? So maybe this solution won't need to last long! 🤞

[FrancescoBorzi]

This looks an important issue, and becomes more critical as your modules grow.

Ideally it should just be possible to run the TestBed.configureTestingModule inside a beforeAll.

That would be the most elegant solution.

[davidjpfeiffer]

The blog post below explains how to skip the recompilation step so that you can use real dependency injection, compile the component once, and then run all the tests. Thank you to Nikita Yakovenko for sharing this solution!

https://blog.angularindepth.com/angular-unit-testing-performance-34363b7345ba.

[kekel87]

Yep, @davidjpfeiffer, as I mentioned above, this blog post update speak about ng-bullet.

Moreover almost all the projects of my teams have been there for almost a year, and it still works very well! Combined with ng-mock, It really does good jobs !

[DenysVuika]

What is more disturbing is that this problem with TestBed has been there for years already.

[MatMercer]

We have more than 600 tests in our project and it takes more than 5 minutes to execute everything. Or 3 minutes to start a simple test. I agree with @brian428 "It basically cancels out the whole point of TDD.".

Working this way is very very bad.

[Goodwine]

@MatMercer something that really helped me was a "recent" Angular update that lets you specify the flag --include, so during development I set up something like:

git diff HEAD^ --name-only | xargs -L 1 dirname | sort | uniq etc and eventually got the output to be something like {dir1/a/b/*.ts,dir2/a/c/*ts} which I passed to ng test --include ${test_glob?}.

It's ugly, but you can always call ng test --include dir/i/am/working/on/ too

I was able to move faster (especially combined with our fork of ng-bullet). And like others have mentioned, karma-parallel also helped bring down 4k tests from 20min to 5min on the full test suite

[MatMercer]

@Goodwine I'm currently using angular 5.2.0. ng-bullet for some reason causes problems with npm (the o'l peer dependencies problem). About the karma-parallel, it only bring it down from 5:20 to 4:40, not a huge difference. I tried using your suggestion, but since you already said

what really helped me was a "recent" Angular update

Since I'm using 5.2.0, I don't have the "recent" features.

About the other solutions presented here, I can't use them, since the tests changes the components/use spy a lot. I really don't know what else to do from here. Maybe upgrade the Angular version and pray for it to work?

[giniedp]

you dont need the ng-bullet dependency. The implementation is pretty simple.

this is what we use in our project

import { getTestBed, TestBed } from "@angular/core/testing"

export function configureTestSuite(configureAction: () => Promise<any>) {
  const testBedApi = getTestBed()
  const originReset = TestBed.resetTestingModule
  beforeAll(() => {
    TestBed.resetTestingModule()
    TestBed.resetTestingModule = () => TestBed
  })
  if (configureAction) {
    beforeAll((done) => {
      configureAction().then(done).catch(done.fail)
    })
  }
  afterEach(() => {
    testBedApi["_activeFixtures"].forEach((fixture) => fixture.destroy())
    testBedApi["_instantiated"] = false
  })
  afterAll(() => {
    TestBed.resetTestingModule = originReset
    TestBed.resetTestingModule()
  })
}

with that you have no peer dependency problems. Further you are in control of the behavior so you can revert it at any time, without changing your tests.

[Goodwine]

@MatMercer

Since I'm using 5.2.0, I don't have the "recent" features.

Be aware that angular versions are only supported for a limited amount of time [docs]
And also, if the Angular team improves the test setup performance like you want, it would be on a
newer version, so you would have to upgrade anyways, there's really not workaround*.

On another Issue about something similar, someone mentioned that the next Angular release with Ivy and R3SomethingTestBed made test setup run much faster even without AOT.
angular/angular-cli#6650

• I found a workaround, it's not pretty, basically you use something like git diff HEAD^ --name-only and process the data to generate something that you can pass to const context = require.context(... in the test.ts file, but that means you have to be regenerating the HTML file every time (which I was doing using a gulp task watching changes for TS/HTML files)

(Disclaimer: I'm not a member of Angular team)

[MatMercer]

I'm currently refactoring my component tests (49 files) to use ng-bullet that I've added manually from here into my project as @giniedp said. The snippet @giniedp gave is a little bit outdated so I'm giving a warning to the newcomers: use the ng-bullet from here, it's updated and worked with my angular 5.2.0.

@Goodwine Indeed, what you have said is right, if they will give performance improvements in the next releases there is no "workarounds", we will need to upgrade it anyway.

After I finish the refactor I'll share the results here, I'll probably use karma-parallel too.

[giniedp]

it is actually the very same implementation as the latest version of ng-bullet, except that i choose to skip await TestBed.compileComponents().

[MatMercer]

@giniedp

The signature of your snippet that gave me compilation errors:

export function configureTestSuite(configureAction: () => Promise<any>) {

The signature of the ng-bullet project:

export const configureTestSuite = (configureAction?: () => void) => {

Probably the way we used them in our tests is different (I'm using based in the ng-bullet npm page), that's why your snippet didn't worked for me.

[giniedp]

you are right. That is another minor difference in how we adopted from ng-bullet. The change on your side would have been

-   configureTestSuite(() => {
+   configureTestSuite(async () => {
        TestBed.configureTestingModule({
            // ...
        })
    });

for us it was important not to take away the flow control from the developer, so he is still able to do async stuff inside configureTestSuite

    configureTestSuite(async () => {
        await TestBed.configureTestingModule({
            // ...
        }).compileComponents()
        await moreAsyncStuff()
    });

[MatMercer]

@giniedp @Goodwine

That's interesting @giniedp. In my use-case we don't care about the flow control.

I have to say thanks! I successfully added ng-bullet in the project and the tests execution went from 5m:30s to 30s. The problem I had with the peer dependencies before was because my node version was too outdated.

Adding karma-parallel didn't did much (28s instead of 30s) and broke a test so I didn't used it.

[Quramy]

@Goodwine

On another Issue about something similar, someone mentioned that the next Angular release with Ivy and R3SomethingTestBed made test setup run much faster even without AOT.

Yes. enabledIvy: true (it's tsconfig.json's option, see https://angular.io/guide/ivy#opting-into-angular-ivy) configures to use TestBedRender3 as the TestBed implementation. It's TestBed for Render 3, a.k.a. Ivy. And Ivy JiT compiler has caching mechanism for compiled modules and components. ( See https://github.com/angular/angular/blob/8.2.14/packages/core/src/render3/jit/module.ts#L49-L66 ).

So if you turn Ivy on, TestBed.configureTestingModule(...).compileComponents() gets faster because the second or later compilation for the same components are skipped.

[JoostK]

With Ivy, the TestBed infrastructure has been overhauled and no longer requires recompilation of testing modules. I have seen significant speedups (8x) in tests because of this change.

[krokofant]

@JoostK Good to hear!

---

Would nice to have an overview of the performance differences between:

1. Angular 8 with Karma
2. Angular 8 with Jest
3. Angular 9 with Karma (+Ivy)
4. Angular 9 with Jest (+Ivy)

[AndrewKushnir]

Hello everyone,

As @JoostK mentioned, Ivy TestBed was overhauled to avoid unnecessary recompilations between tests, which helped improve overall performance. We did some measurements to compare ViewEngine and Ivy TestBed versions using Material Components test suite, Angular.io app and Angular Framework unit tests - performance improvement was around 30-40%. We also received feedback from developers that they see even better results on their projects.

I’m closing this ticket and feel free to create new ones if you notice TestBed performance issues after switching to Ivy.

Thank you.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}