@angular-devkit/build-angular:jest Memory Leak and slower test execution speed on CI

[katedoctor]

Command

test

Is this a regression?

• Yes, this behavior used to work in the previous version

The previous version in which this bug was not present was

15

Description

Would someone be able to help me?

I migrated to Angular 16, and to use es modules, I migrated all tests to Jest.
After hundreds of attempts to configure tests ( by the way, @angular-devkit/build-angular:jest + 'jest-preset-angular' doesn't work) I faced 3 main errors in a unit test.

№ 1
TypeError: window.matchMedia is not a function

 38 | };
      39 | var __commonJS = (cb, mod) => function __require() {
    > 40 |   return mod || (0, cb[__getOwnPropNames(cb)[0]])((mod = { exports: {} }).exports, mod), mod.exports;
         |                                                  ^
      41 | };
      42 | var __export = (target, all) => {
      43 |   for (var name in all)

      at mediaMatch (node_modules/tinymce/tinymce.js:957:40)
      at mediaMatch (node_modules/tinymce/tinymce.js:615:35)
      at Object.detect$3 [as detect] (node_modules/tinymce/tinymce.js:948:26)
      at node_modules/tinymce/tinymce.js:958:53
      at node_modules/tinymce/tinymce.js:605:17
      at detect$2 (node_modules/tinymce/tinymce.js:959:28)
      at node_modules/tinymce/tinymce.js:962:24
      at node_modules/tinymce/tinymce.js (node_modules/tinymce/tinymce.js:31756:1)
      at __require (chunk-ATHAJZBS.mjs:40:50)
      at src/app/survey-builder/components/survey-questions/components/html-editor/html-editor.component.ts:6:21

component.spec

describe('HtmlEditorComponent', () => {
  let component: HtmlEditorComponent;
  let fixture: ComponentFixture<HtmlEditorComponent>;

  beforeAll(() => {
    jest.mock('tinymce', () => ({
      init: jest.fn()
    }));

    Object.defineProperty(window, 'matchMedia', {
      writable: true,
      value: {
        matches: false,
        media: query,
        onchange: null,
        addListener: jest.fn(),
        removeListener: jest.fn()
      }
    });
  });

  beforeEach(waitForAsync(() => {
    TestBed.configureTestingModule({
      declarations: [HtmlEditorComponent],
      providers: [
        {
          provide: SurveyBuilderHelperService,
          useValue: {
            questionReorderInProgress: new EventEmitter()
          }
        }
      ]
    }).compileComponents();
  }));

  beforeEach(() => {
    fixture = TestBed.createComponent(HtmlEditorComponent);
    component = fixture.componentInstance;
    fixture.detectChanges();
  });

  afterEach(() => {
    fixture.destroy();
  });

  it('should create', () => {
    expect(component).toBeTruthy();
  });
});

№ 2
NG0303: Can't bind to 'ngIf' since it isn't a known property of 'div' (used in the '_a' component template).
component.html:

<div class="question"
     *ngIf="!!group.id"
     [class.hidden]="group.show === false"
     [class.branching]="!!group.branching_rule">
  <div class="question-content">
  content
</div>
</div>

and component.spec:

describe('MatrixComponent', () => {
  let component: MatrixComponent;
  let fixture: ComponentFixture<MatrixComponent>;

  beforeEach(waitForAsync(() => {
    TestBed.configureTestingModule({
      imports: [FormsModule, ReactiveFormsModule, CommonModule],
      declarations: [
        QuestionComponent,
        MockComponent(BranchRuleComponent),
        MockComponent(ButtonComponent),
        MockComponent(IconComponent),
        MockComponent(TitleComponent),
        MockComponent(TooltipComponent),
        MockComponent(GridMenuComponent),
        MockComponent(ToggleComponent),
        MockComponent(QuestionSidebarComponent),
        MockDirective(InputDirective)
      ],
      providers: [
        {
          provide: ModalService,
          useValue: jest.fn()
        },
        {
          provide: SurveyBuilderApiService,
          useValue: jest.fn()
        },
        {
          provide: SurveyBuilderLanguageService,
          useValue: jest.fn()
        },
        {
          provide: SurveyBuilderModalService,
          useValue: {
            showQuestionOrderModal: jest.fn().mockReturnValue({})
          }
        }
      ]
    }).compileComponents();
  }));

  beforeEach(() => {
    fixture = TestBed.createComponent(MatrixComponent);
    component = fixture.componentInstance;
    fixture.detectChanges();
  });

  it('should create', () => {
    expect(component).toBeTruthy();
  });
});

№ 3
NG0201: No provider for NgControl found in NodeInjector

component.spec

describe('CategoryAttachQuestionsModalComponent', () => {
  let component: CategoryAttachQuestionsModalComponent;
  let fixture: ComponentFixture<CategoryAttachQuestionsModalComponent>;

  beforeEach(async () => {
    await TestBed.configureTestingModule({
      imports: [RouterTestingModule, NzDropDownModule, ReactiveFormsModule],
      declarations: [CategoryAttachQuestionsModalComponent, MockDirective(NzInputDirective), MockComponent(NzInputGroupComponent)],
      providers: [
        {
          provide: ModalService,
          useValue: jest.fn()
        },
        {
          provide: NzModalRef,
          useValue: {
            destroy: jest.fn().mockReturnValue(of())
          }
        },
        {
          provide: CategoryManagementService,
          useValue: {
            searchSimpleQuestion: jest.fn().mockReturnValue(of({})),
            addQuestionsToCategory: jest.fn().mockReturnValue(of({}))
          }
        },
        {
          provide: ToastService,
          useValue: jest.fn()
        }
      ]
    }).compileComponents();

    fixture = TestBed.createComponent(CategoryAttachQuestionsModalComponent);
    component = fixture.componentInstance;
    fixture.detectChanges();
  });

  it('should create', () => {
    expect(component).toBeTruthy();
  });
});

test config:

angular.json

 "test": {
          "builder": "@angular-devkit/build-angular:jest",
          "options": {
            "tsConfig": "tsconfig.spec.json",
            "polyfills": [
              "zone.js",
              "zone.js/testing"
            ]
          }

package.json:

{
type: "module",
  "dependencies": {
    "@angular/animations": "^16.2.12",
    "@angular/common": "^16.2.12",
    "@angular/compiler": "^16.2.12",
    "@angular/core": "^16.2.12",
    "@angular/forms": "^16.2.12",
    "@angular/language-service": "^16.2.10",
    "@angular/platform-browser": "^16.2.12",
    "@angular/platform-browser-dynamic": "^16.2.12",
    "@angular/platform-server": "^16.2.10",
    "@angular/router": "^16.2.12",
    "@cypress/schematic": "^1.3.0",
    "@ngrx/effects": "16.3.0",
    "@ngrx/store": "16.3.0",
    "@ngrx/store-devtools": "16.3.0",
    "@sentry/browser": "5.7.0",
    "angular-code-input": "2.0.0",
    "avid-ui-lib": "1.100.3",
    "big.js": "^5.2.2",
    "chart.js": "4.4.0",
    "chartjs-adapter-dayjs-4": "^1.0.4",
    "chartjs-plugin-datalabels": "^2.2.0",
    "core-js": "^3.30.1",
    "d3": "^7.6.1",
    "dayjs": "^1.11.10",
    "dom-autoscroller": "^2.3.4",
    "dragula": "^3.7.3",
    "handorgel": "^0.4.9",
    "js-sha256": "^0.10.1",
    "ng-zorro-antd": "16.2.1",
    "ng2-dragula": "5.0.1",
    "ngx-infinite-scroll": "16.0.0",
    "ngx-mask": "16.3.9",
    "ramda": "^0.28.0",
    "rxjs": "^6.5.5",
    "tinymce": "6.8.2",
    "tslib": "^2.0.0",
    "url-search-params-polyfill": "^5.1.0",
    "webpack": "^4.29.6",
    "zone.js": "0.13.3"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "^16.2.7",
    "@angular-eslint/builder": "16.2.0",
    "@angular-eslint/eslint-plugin": "16.2.0",
    "@angular-eslint/eslint-plugin-template": "16.2.0",
    "@angular-eslint/schematics": "16.2.0",
    "@angular-eslint/template-parser": "16.2.0",
    "@angular/cli": "^16.2.7",
    "@angular/compiler-cli": "^16.2.10",
    "@babel/core": "^7.23.5",
    "@babel/preset-env": "^7.23.5",
    "@babel/preset-typescript": "^7.23.3",
    "@digitalroute/cz-conventional-changelog-for-jira": "^7.4.0",
    "@types/big.js": "^4.0.5",
    "@types/jest": "^29.5.11",
    "@types/node": "^20.8.9",
    "@types/ramda": "^0.28.20",
    "@typescript-eslint/eslint-plugin": "5.44.0",
    "@typescript-eslint/parser": "6.13.1",
    "commitizen": "^4.2.5",
    "cypress": "7.5.0",
    "eslint": "^8.28.0",
    "eslint-plugin-log": "^1.2.7",
    "husky": "^8.0.3",
    "jest": "^29.7.0",
    "jest-environment-jsdom": "^29.7.0",
    "jest-preset-angular": "^13.1.5",
    "jsdom": "^23.2.0",
    "lint-staged": "^13.1.0",
    "ng-mocks": "^14.10.0",
    "node-sass": "8.0.0",
    "prettier": "^2.0.5",
    "ts-node": "^10.9.2",
    "typescript": "5.1.6"
  },
}

Minimal Reproduction

I have Tinymce lib which is using window.matchMedia function. I tried to mock it with jest, but it didn't work.

Exception or Error

No response

Your Environment

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1602.11
@angular-devkit/build-angular   16.2.11
@angular-devkit/core            16.2.11
@angular-devkit/schematics      12.2.18
@angular/cli                    16.2.11
@schematics/angular             16.2.11
rxjs                            6.6.7
typescript                      5.1.6
webpack                         4.47.0
zone.js                         0.13.3

Anything else relevant?

Also, there is a big problem that @angular-devkit/build-angular:jest doesn't allow to run single test!
Also I can't run tests when I have symlink.

[dgp1130]

Taking a quick look at these:

@angular-devkit/build-angular:jest + 'jest-preset-angular' doesn't work

You shouldn't need jest-preset-angular. @angular-devkit/build-angular:jest is intended to handle all the configuration for you.

TypeError: window.matchMedia is not a function

I'm guessing Jest doesn't implement matchMedia given that it's not running a real browser and doesn't process CSS. You'll either need to not call that function in a test or mock it. It looks like you're trying to mock it, but matchMedia is a function, not an object.

https://developer.mozilla.org/en-US/docs/Web/API/Window/matchMedia

I think you want something like:

window.matchMedia = jest.fn();

Can't bind to 'ngIf' since it isn't a known property of 'div' (used in the '_a' component template).

This sounds like NgIf isn't being imported correctly. I see imports: [CommonModule] which should include that. I assume QuestionComponent is the component with the template? If you can create a minimal reproduction of this error, we can investigate further.

No provider for NgControl found in NodeInjector

I'm not very familiar with forms, but this also sounds like an import issue. I think you might need to import FormsModule for this. It's hard to debug further without more context though. A minimal reproduction would help us investigate further.

Also, there is a big problem that @angular-devkit/build-angular:jest doesn't allow to run single test!
Also I can't run tests when I have symlink.

Jest support is still experimental, and these just aren't features we've gotten to yet. You can file separate issues for these (I don't think either of these have been filed already) to make sure they're looked at before the builder is marked stable.

[katedoctor]

@dgp1130 thanks.

I successfully resolved the errors, but I still have some questions.

Initially, window.matchMedia = jest.fn(); didn't work as expected because all such mocks need to be set up before the test starts. To address this, I created a setup file with the following logic:

Object.defineProperty(window, 'matchMedia', {
writable: true,
value: jest.fn().mockImplementation((query) => ({
matches: false,
media: query,
onchange: null,
addListener: jest.fn(), // deprecated
removeListener: jest.fn(), // deprecated
addEventListener: jest.fn(),
removeEventListener: jest.fn(),
dispatchEvent: jest.fn()
}))
});

This setup file is referenced in jest.config like so:

const config: Config = {
setupFiles: [../../setup.jest.ts] // TODO: Fix path. This is specific for local run of npm run test
};

export default config;

However, the main challenge is that Jest and @angular-devkit/build-angular:jest use different rootDir settings. For Jest, the rootDir is my project folder, while for @angular-devkit/build-angular:jest, it's project folder/dist/test-out. This discrepancy is the reason for the ../../ in the setupFiles path.

I've encountered a new issue that wasn't present with Karma – a memory leak and slower test execution speed on CI.

Observation:

• The same tests that took 7 minutes with Karma now take 11 minutes using @angular-devkit/build-angular:jest.

Questions:

1. Does @angular-devkit/build-angular:jest support options like --shard or --runInBand to optimize test execution time?
2. How can I address the error "FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory" that occurs during testing?

These concerns weren't present with Karma, indicating a potential issue with Jest or its integration with Angular.

EDIT:
all tests even default ones are claimed as memory leaks when "detectLeaks: true" is set

[dgp1130]

This setup file is referenced in jest.config like so:

Currently we're exploring not exposing the underlying Jest configuration, so this shouldn't be possible. There's a bug right now where it does load config files, which is incorrect and I need to fix that at some point.

I don't think mocking matchMedia should require a config option. I would expect beforeEach / afterEach to work here:

describe('my-test', () => {
  beforeEach(() => {
    Object.defineProperty(window, 'matchMedia', { /* ... */ });
  });

  afterEach(() => {
    delete window.matchMedia;
  });

  it('uses matchMedia', () => {
    window.matchMedia(/* ... */);
  });
});

I've encountered a new issue that wasn't present with Karma – a memory leak and slower test execution speed on CI.
How can I address the error "FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory" that occurs during testing?
all tests even default ones are claimed as memory leaks when "detectLeaks: true" is set

Where are you seeing this error? Is it the Jest process or Angular CLI process? If you can provide a minimal reproduction I'd be interested in taking a look. Can you share the full output where you're seeing the test identified by detectLeaks?

These concerns weren't present with Karma, indicating a potential issue with Jest or its integration with Angular.

Regarding comparison with Karma, Jest is not intended to be a 1:1 replacement for Karma. Jest will have a very different performance profile and may be faster or slower in some instances. I don't think the experiment is far enough along to know how they actually compare in practice. Web Test Runner is intended to be a replacement for Karma and is more likely to have directly improved performance.

Does @angular-devkit/build-angular:jest support options like --shard or --runInBand to optimize test execution time?

Not currently, we're still early in this experiment. You're welcome to file issues more directly focused on those options. Though I don't think --runInBand is intended to improve performance, I was under the impression it was intended for debuggability.

[katedoctor]

@dgp1130

I don't think mocking matchMedia should require a config option.

I share your thoughts on the approach, but unfortunately, using mocks in beforeEach, afterEach, or beforeAll didn't work in my case. The configuration file which initialized before test execution was the only viable solution. I've extensively experimented with adding mocks directly in the spec file, but without success.

As for the memory leaks, I enabled 'detectLeaks' in Jest, and it flagged all my tests with leak issues. This output highlights a significant problem:

describe('SurveyLanguageComponent', () => {
  let component: SurveyLanguageComponent;
  let fixture: ComponentFixture<SurveyLanguageComponent>;

  beforeEach(
    waitForAsync(() => {
      TestBed.configureTestingModule({
        declarations: [
          SurveyLanguageComponent,
          MockComponent(GroupTranslateComponent),
          MockComponent(TextTranslateComponent),
          MockComponent(LoadingSpinnerComponent),
          MockComponent(CheckboxComponent),
          MockComponent(IconComponent)
        ],
        providers: [
          {
            provide: SurveyBuilderHelperService,
            useValue: {
              survey: {}
            }
          },
          {
            provide: SurveyBuilderApiService,
            useValue: jest.fn()
          },
          {
            provide: SurveyBuilderLanguageService,
            useValue: {
              fetchLocales: jest.fn().mockReturnValue(of({ items: [] }))
            }
          },
          {
            provide: ToastService,
            useValue: jest.fn()
          }
        ]
      }).compileComponents();
    })
  );
beforeEach(() => {
    fixture = TestBed.createComponent(SurveyLanguageComponent);
    component = fixture.componentInstance;
    component.languageHelper.supportedLocales = [];
    fixture.detectChanges();
  });

  it('should create', () => {
    expect(component).toBeTruthy();
  });
});

and error:

dist/test-out/survey-language.component.spec.mjs
 ● Test suite failed to run

   EXPERIMENTAL FEATURE!
   Your test suite is leaking memory. Please ensure all references are cleaned.

   There is a number of things that can leak memory:
     - Async operations that have not finished (e.g. fs.readFile).
     - Timers not properly mocked (e.g. setInterval, setTimeout).
     - Keeping references to the global scope.

I'm aware that the @angular-devkit/build-angular:jest builder is experimental, but currently, I'm unable to run tests effectively using Jest. I've opened an issue with Jest or angular-preset regarding zone.js and Jest compatibility ([https://github.com/thymikee/jest-preset-angular/issues/2211]). Since Karma is deprecated and doesn't support ES6 modules – which are necessary due to some packages being available only as ES6 modules – I'm exploring alternative solutions.

I mean, some options to speed up process would be nice. Now each test run file by file and not in parallel.

[dgp1130]

I share your thoughts on the approach, but unfortunately, using mocks in beforeEach, afterEach, or beforeAll didn't work in my case. The configuration file which initialized before test execution was the only viable solution. I've extensively experimented with adding mocks directly in the spec file, but without success.

That's surprising to me and makes me think your test might be calling matchMedia before it is mocked (maybe as a top-level statement of a module?). If you can create a minimal reproduction, I can take a look.

As for the memory leaks, I enabled 'detectLeaks' in Jest, and it flagged all my tests with leak issues.

I'm able to reproduce this one. However I'm also able to reproduce this for an empty test.

describe('test', () => {
  it('runs', () => {
    expect(true).toBeTrue();
  });
});

This fails with detectLeaks even when running directly with Jest and not involving Angular CLI at all. If I run with CommonJS, it passes, but ESM consistently fails. Given that this option is experimental, it may have it's own stability problems with ESM, but I don't see an Angular bug here right now.

I mean, some options to speed up process would be nice. Now each test run file by file and not in parallel.

Maybe I'm misunderstanding but I'm pretty sure our Jest usage is already parallelized? If you create two test files which wait 10 seconds, the total execution time is ~10 seconds, not 20, so they are running parallel.

Since Karma is deprecated and doesn't support ES6 modules – which are necessary due to some packages being available only as ES6 modules

I was under the impression our Karma builder bundles with Webpack, which should support ESM and CommonJS dependencies. I feel like this should be feasible with our Karma setup.

[katedoctor]

@dgp1130

That's surprising to me and makes me think your test might be calling matchMedia before it is mocked

Yes, as mentioned earlier, the issue with matchMedia arises because it's invoked by node_modules packages like TinyMCE before it gets mocked. Attempts to mock TinyMCE directly were also unsuccessful. I recall coming across a Jest discussion indicating that such mocks need to be set up before the tests run, which aligns with the solution of using a config file.

but I don't see an Angular bug here right now.

Regarding the memory usage with Angular's builder, my experience showed it consuming significantly more memory compared to Jest. A notable difference is that the Angular builder doesn't run tests in parallel, which might contribute to this issue.

I was under the impression our Karma builder bundles with Webpack, which should support ESM and CommonJS dependencies. I feel like this should be feasible with our Karma setup.

The moment when I upgraded Angular from version 15 to 16, our tests started failing with the error: "cannot use import statement outside a module". If you have any insights or examples of how Karma should be configured in Angular to address this, it would be immensely helpful. I didn’t find specific instructions in the Angular upgrade guide, but there's a growing trend of moving towards Jest in the community.

[alan-agius4]

Hi @katedoctor,

Regarding the memory usage with Angular's builder, my experience showed it consuming significantly more memory compared to Jest. A notable difference is that the Angular builder doesn't run tests in parallel, which might contribute to this issue.

When using the Angular the Jest builder, tests are executed in parallel, The difference in consumption in memory can be attributed to various things, it would definitely help if you can provide a reproduction so that we can take a look and compare it with a plain Jest setup that you mentioned. In the CLI builder, we do use --experimental-vm-modules when running Jest, which uses native ESM modules instead of transforming them to CJS. This can in some cases cause memory leaks in Jest see: jestjs/jest#14605

"cannot use import statement outside a module".

This typically indicates that one or more files are not being processed by Webpack, unfortunately without a minimal reproduction it's hard to provide a solution.

[katedoctor]

@alan-agius4

I guess, maybe. But nothing was changed except angular version. I couldn't find anything to change, as tests were successful but in the end got error:

Chrome Headless 120.0.6099.216 (Mac OS 10.15.7) ERROR
An error was thrown in afterAll
Uncaught SyntaxError: Cannot use import statement outside a module
SyntaxError: Cannot use import statement outside a module

When using the Angular the Jest builder, tests are executed in parallel

My apologies for any confusion earlier. I have a hypothesis about the slower performance. When running tests with Jest, I've noticed it processes files in chunks.

However, with Angular, the process is different – it handles files one by one. Then it ends with: "FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory". This leads me to believe that the way Angular processes test files might be different.

[dgp1130]

I don't think we can help more here without minimal reproductions. If you can provide a reproduction of the Karma problem (would be better in a separate issue), or of the out of memory error then we can look at those individually.

[katedoctor]

@dgp1130

While working on a step-by-step project with Karma, I identified the root cause of Karma problem. It turns out the issue was linked to scripts included in angular.json for integrating TinyMCE into our build or tests. It seems that Karma struggled to process these scripts, leading to the error: "Cannot use import statement outside a module." It wasn't immediately obvious to me since those scripts have been part of angular.json forever.

[dgp1130]

It sounds like the relevant issues have been resolved so I'm going to close this issue. If you find some broken behavior and are able to create a minimal reproduction, please file a new issue and we can investigate further.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}