Slower Build Performance after Upgrade from 9.0.7 to 9.1.0

[sbradl]

Since the update from Angular 9.0.7 to 9.1.0 our build times increased a lot. Here are some times:

v9.0.7 - prod build: 8:41
v9.1.0 - prod build: 13:09

These times are on a windows machine with node v12

On linux and mac the difference is much smaller - only a few seconds

[petebacondarwin]

Could you give us some more information about your setup and what 3rd party libraries you are using.
Also from watching the build can you identify what part of the build process has slowed down?

[sbradl]

It is quite a large project.
The build machine has a Core i7 with 4 cores and 32GB of RAM.
3rd party libs are mostly material and a few small libs

One small portion of the build:
v9.0.7 (a bit over 1 minute)
17:46:45
Compiling TypeScript sources through ngc
17:46:49
Compiling @angular/cdk/keycodes : es2015 as esm2015
17:46:51
Compiling @angular/cdk/observers : es2015 as esm2015
17:46:52
Compiling @angular/cdk/a11y : es2015 as esm2015
17:46:53
Compiling @angular/animations : es2015 as esm2015
17:46:54
Compiling @angular/cdk/portal : es2015 as esm2015
17:46:55
Compiling @angular/platform-browser : es2015 as esm2015
17:46:56
Compiling @angular/animations/browser : es2015 as esm2015
17:46:58
Compiling @angular/platform-browser/animations : es2015 as esm2015
17:46:59
Compiling @angular/forms : es2015 as esm2015
17:47:01
Compiling @angular/material/core : es2015 as esm2015
17:47:02
Compiling @angular/cdk/overlay : es2015 as esm2015
17:47:04
Compiling @angular/material/menu : es2015 as esm2015
17:47:08
Compiling @angular/material/progress-spinner : es2015 as esm2015
17:47:11
Compiling @angular/material/divider : es2015 as esm2015
17:47:15
Compiling @angular/material/icon : es2015 as esm2015
17:47:19
Compiling @angular/material/dialog : es2015 as esm2015
17:47:22
Compiling @ngx-translate/core : es2015 as esm2015
17:47:25
Compiling @angular/material/button : es2015 as esm2015
17:47:29
Compiling @angular/cdk/text-field : es2015 as esm2015
17:47:31
Compiling @angular/material/form-field : es2015 as esm2015
17:47:32
Compiling @angular/material/input : es2015 as esm2015
17:47:36
Compiling @angular/cdk/table : es2015 as esm2015
17:47:38
Compiling @angular/material/select : es2015 as esm2015
17:47:39
Compiling @angular/cdk/layout : es2015 as esm2015
17:47:41
Compiling @angular/material/tooltip : es2015 as esm2015
17:47:43
Compiling @angular/material/paginator : es2015 as esm2015
17:47:44
Compiling @angular/material/sort : es2015 as esm2015
17:47:46
Compiling @angular/material/table : es2015 as esm2015
17:47:49
Compiling angular-split : es2015 as esm2015
17:47:51
Compiling angular-plotly.js : es2015 as esm2015

v9.1.0 (nearly 5 minutes)
10:16:16
Compiling TypeScript sources through ngc
10:17:28
Compiling @angular/cdk/keycodes : es2015 as esm2015
10:17:30
Compiling @angular/cdk/observers : es2015 as esm2015
10:17:31
Compiling @angular/cdk/a11y : es2015 as esm2015
10:17:32
Compiling @angular/animations : es2015 as esm2015
10:17:33
Compiling @angular/cdk/portal : es2015 as esm2015
10:17:34
Compiling @angular/platform-browser : es2015 as esm2015
10:17:35
Compiling @angular/animations/browser : es2015 as esm2015
10:17:37
Compiling @angular/platform-browser/animations : es2015 as esm2015
10:17:38
Compiling @angular/forms : es2015 as esm2015
10:17:40
Compiling @angular/material/core : es2015 as esm2015
10:17:42
Compiling @angular/cdk/overlay : es2015 as esm2015
10:17:44
Compiling @angular/material/menu : es2015 as esm2015
10:17:52
Compiling @angular/material/progress-spinner : es2015 as esm2015
10:17:59
Compiling @angular/material/divider : es2015 as esm2015
10:18:44
Compiling @angular/material/icon : es2015 as esm2015
10:18:53
Compiling @angular/material/dialog : es2015 as esm2015
10:19:00
Compiling @ngx-translate/core : es2015 as esm2015
10:19:08
Compiling @angular/material/button : es2015 as esm2015
10:19:16
Compiling @angular/cdk/text-field : es2015 as esm2015
10:19:18
Compiling @angular/material/form-field : es2015 as esm2015
10:19:20
Compiling @angular/material/input : es2015 as esm2015
10:19:28
Compiling @angular/cdk/table : es2015 as esm2015
10:19:30
Compiling @angular/material/select : es2015 as esm2015
10:19:32
Compiling @angular/cdk/layout : es2015 as esm2015
10:19:33
Compiling @angular/material/tooltip : es2015 as esm2015
10:19:35
Compiling @angular/material/paginator : es2015 as esm2015
10:19:37
Compiling @angular/material/sort : es2015 as esm2015
10:19:38
Compiling @angular/material/table : es2015 as esm2015
10:20:27
Compiling @angular/cdk/tree : es2015 as esm2015
10:20:36
Compiling @angular/material/tree : es2015 as esm2015
10:20:43
Compiling angular-split : es2015 as esm2015
10:20:50
Compiling angular-plotly.js : es2015 as esm2015
10:20:59
Compiling @angular/material/checkbox : es2015 as esm2015
10:21:07
Compiling @angular/material/button-toggle : es2015 as esm2015

So the first step "Compiling TypeScript sources through ngc" increased quite a bit from 4s to over a minute and the rest also increased.

[sbradl]

Another thing: Our projects consists of multiple libraries. The output from above is from one of the larger ones which depends on some of our other libs.

[blackholegalaxy]

I also encounter this behavior with a 30% build time increase. mainly due to the ngcc compiling. The ngcc compiling is far from the concurrent run announced in 9.1.0. They ALL run in sequence. Even on big machines with 6, 12 or 18 CPUS (and double threads). Having material in the projet makes the build time explode.

Is there any option to "activate" ngcc concurrent compile? I'd expect ngcc to spawn many "runners" depending on CPU and threads of the machine then batch compile everything to really speedup things. Please note we don't use ngcc postinstall script. We rely only on ngcc ran at ng build time.

As of information / repro: we use angular raw with @angular/material components.

Fun fact: adding a new dependency in the project (dotenv) caused the entire ngcc compile to occur again even if it was the only changed dependency.

[petebacondarwin]

In 9.1.0 the async ngcc processing should be running by default. I am not sure why it would then need to run sync to process things like material during the compile. Perhaps @alan-agius4 has some insight.

I think we should move this discussion to the CLI repository, since it looks like it is the async CLI+ngcc integration that is causing the issue. Transferring...

[sbradl]

Are you sure it is an ngcc problem? The first line in my output is the normal ngc step and even this increased a lot

[alan-agius4]

The sync NGCC is run because of possible modules added in the compilation at a later stage after the first build. Example, when doing code changes, in a local library or adding a new NPM module.

That being said, if a module was already processed by the async NGCC should bail out pretty fast, it will just check for the build marker and exit.

---

@blackholegalaxy

Fun fact: adding a new dependency in the project (dotenv) caused the entire ngcc compile to occur again even if it was the only changed dependency.

This is expected if you use Yarn as your package manager. This is because Yarn will remove the entire node_modules and re-install all dependencies when you add a new dependency. Which means that NGCC will need to re-run on all modules, which in this case will only run in sync mode.

@blackholegalaxy, can you please share the output of ng version also can you explain how are you checking if NGCC is being run in parallel or not?

[alan-agius4]

@petebacondarwin / @sbradl

There are quite a number of reasons why this is slower at least from what I see from the logs and knowing how NGCC works;

1. The first part of NGCC did indeed increase to about a minute. In version 9.1, this part is were node_module tree is analysed to determine which libraries are compiled by Angular and needs to be transformed by NGCC.
2. We are compiling all Angular entry points even those that will never be used when using the Async mode.
3. Spawning a cluster worker is expensive and most likely the regression to process each entrypoint that we are seeing.
4. There might be a bottleneck on IO when running workers because of a lot of disk reads and writes

[sbradl]

With angular/angular-cli#1 you mean the ngc step? This is the biggest problem for us because this also happens when we develop with --watch.

angular/angular-cli#4 could be. The windows system I used has an old ssd and my linux system has a new m2 ssd.

[alan-agius4]

With #1 you mean the ngc step? This is the biggest problem for us because this also happens when we develop with --watch.

Yes, me and @petebacondarwin discussed this in the past and @petebacondarwin had some ideas on how to speed this up when there is no work to be done. But it involved quite a decent amount of work at the time.

#4 could be. The windows system I used has an old ssd and my linux system has a new m2 ssd.

From my experience in general IO operations are always slower on Windows.

[petebacondarwin]

So the actual ngcc processing should still only run once per package, so unless you modify your node_modules, the processing of these packages should not happen a second time.

Regarding 1. - this might well be the problem, since on Windows it is quite slow to traverse the file-system checking for entry-points that "might" need processing. This is happening even if it turns out that they are all already processed. That being said, I recently implemented an __ngcc_entry_point_manifest__.json file that should be written to the top of your node_modules directory that should avoid having to do this traversal... Can you check to see if one of these is appearing?

[petebacondarwin]

Oh, the other part of 1. is that once we have the list of entry-points, which can come from the manifest, we are still parsing the entry-points looking for dependencies in order to sort them before we determine whether they need to be processed or not. This is what @alan-agius4 is talking about here:

Yes, me and @petebacondarwin discussed this in the past and @petebacondarwin had some ideas on how to speed this up when there is no work to be done. But it involved quite a decent amount of work at the time.

Perhaps now is the time to fix this?