Upgrading from v9 to v11 ng build/serve are very slow #20227

[MathewBerg]

🐞 Bug report

Command (mark with an x)

• new
• build
• serve
• test
• e2e
• generate
• add
• update
• lint
• extract-i18n
• run
• config
• help
• version
• doc

Is this a regression?

Maybe? Hard to tell if we just aren't doing something right currently.

Description

We were unable to upgrade from angular 9 to 11 last year. After doing the migration the builds are taking around an hour and ng serve is taking around 10minutes to start and then a few minutes to recompile.

🔬 Minimal Reproduction

The project is quite large. After the discussion here: #20227 we had lot of failures of generating the .json files with NG_BUILD_PROFILING set to 1. Eventually one of our developers was able too after cutting out a large portion of our code base, but it just mimics how we're building stuff elsewhere.

🔥 Exception or Error

(copied from the last bug) I saw this issue raised: #17557 that was able to spit out the webpack timings but I tried the same command and didn't get anything. Here are some logs from that command though:

Hash: 10062e72e655a942dd34
Version: webpack 4.44.2
Time: 708716ms
Built at: 03/09/2021 10:15:12 AM

LOG from webpack.buildChunkGraph.visitModules
prepare: 157.7756ms
visiting: 11.9087ms
calculating available modules: 0.9064ms
merging available modules: 0.1087ms
visiting: 47.7026ms
calculating available modules: 1.9921ms
merging available modules: 0.023ms
visiting: 1.39ms

Initial Chunk Files | Names | Size
main-es5.3ac0413d5fda208dcf49.js | main | 3.60 MB
main-es2015.3ac0413d5fda208dcf49.js | main | 3.32 MB
polyfills-es5.1577f06bbd7cd498d2aa.js | polyfills-es5 | 131.77 kB
styles.088de8f15a462993e614.css | styles | 108.92 kB
polyfills-es2015.603bb5ba1ae3ee6b7388.js | polyfills | 45.98 kB
runtime-es2015.12e587ff0e65ab518ba8.js | runtime | 3.40 kB
runtime-es5.12e587ff0e65ab518ba8.js | runtime | 3.40 kB

🌍 Your Environment

Angular CLI: 11.2.3
Node: 14.16.0
OS: win32 x64

Angular: 11.2.4
... animations, common, compiler, compiler-cli, core, forms
... language-service, localize, platform-browser
... platform-browser-dynamic, router
Ivy Workspace: Yes

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1102.3 (cli-only)
@angular-devkit/build-angular   0.1102.3
@angular-devkit/core            11.2.3 (cli-only)
@angular-devkit/schematics      11.2.3
@angular/cdk                    11.2.3
@angular/cli                    11.2.3
@angular/flex-layout            11.0.0-beta.33
@angular/material               11.2.3
@schematics/angular             11.2.3
@schematics/update              0.1102.3
ng-packagr                      11.2.4
rxjs                            6.5.5
typescript                      4.0.7

Anything else relevant?

Here's the speed-measure-plugin.json:

{
  "misc": {
    "compileTime": 485627
  },
  "plugins": {
    "AngularCompilerPlugin": 253168,
    "BuildOptimizerWebpackPlugin": 61121,
    "ContextReplacementPlugin": 79,
    "CopyPlugin": 67,
    "MiniCssExtractPlugin": 29,
    "ProfilingPlugin": 70949,
    "ProgressPlugin": 11394,
    "CircularDependencyPlugin": 6080,
    "LicenseWebpackPlugin": 9014,
    "AnyComponentStyleBudgetChecker": 241,
    "SuppressExtractedTextChunksWebpackPlugin": 125,
    "NgBuildAnalyticsPlugin": 5058,
    "HashedModuleIdsPlugin": 631,
    "OptimizeCssWebpackPlugin": 5030,
    "TerserPlugin": 44879,
    "BundleBudgetPlugin": 970
  },
  "loaders": {
    "build": [
      {
        "averages": {
          "dataPoints": 1,
          "median": 202470,
          "mean": 202470,
          "range": {
            "start": 202470,
            "end": 202470
          }
        },
        "activeTime": 202470,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\mini-css-extract-plugin\\dist\\loader.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\css-loader\\dist\\cjs.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\postcss-loader\\src\\index.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\sass-loader\\dist\\cjs.js"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 1,
          "median": 194400,
          "mean": 194400,
          "range": {
            "start": 194400,
            "end": 194400
          }
        },
        "activeTime": 194400,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\css-loader\\dist\\cjs.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\postcss-loader\\src\\index.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\sass-loader\\dist\\cjs.js"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 405,
          "median": 4328,
          "mean": 5250,
          "range": {
            "start": 363,
            "end": 13806
          },
          "variance": 12656309
        },
        "activeTime": 13806,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\raw-loader\\dist\\cjs.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\postcss-loader\\src\\index.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\sass-loader\\dist\\cjs.js"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 1212,
          "median": 5,
          "mean": 26,
          "range": {
            "start": 2,
            "end": 6484
          },
          "variance": 39836
        },
        "activeTime": 32036,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\@angular-devkit\\build-optimizer\\src\\build-optimizer\\webpack-loader.js"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 1304,
          "median": 31,
          "mean": 17,
          "range": {
            "start": 3,
            "end": 644
          },
          "variance": 639
        },
        "activeTime": 22403,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\@angular-devkit\\build-optimizer\\src\\build-optimizer\\webpack-loader.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\@ngtools\\webpack\\src\\index.js"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 3,
          "median": 7,
          "mean": 15,
          "range": {
            "start": 7,
            "end": 29
          },
          "variance": 148
        },
        "activeTime": 45,
        "loaders": [
          "modules with no loaders"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 73,
          "median": 8,
          "mean": 10,
          "range": {
            "start": 7,
            "end": 68
          },
          "variance": 51
        },
        "activeTime": 718,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\svg-inline-loader\\index.js"
        ],
        "subLoadersTime": {}
      }
    ]
  }
}

[clydin]

Is a third-party builder being used within the project?

The version output also may indicate that incorrect/outdated versions are present:

@angular-devkit/architect       0.1102.3 (cli-only)
@angular-devkit/build-angular   0.1102.3
@angular-devkit/core            11.2.3 (cli-only)

A third-party builder may be inadvertently injecting older versions of dependencies into the build pipeline.

It also appears that the build is a production build due to the presence of BuildOptimizerWebpackPlugin. Is that correct?

[MathewBerg]

@clydin yes, this was a production build. We're running this:

node --max_old_space_size=8096 ./node_modules/@angular/cli/bin/ng "build" "--prod"

chrome-profiler-events.json is 600MB

[MathewBerg]

Seeing that sass seems to take awhile, I'm going to try converting as much of the scss we have to css with some scripts.

[clydin]

If you are able to update to v12, the next patch release (12.0.1) will contain several improvements that should greatly benefit the build performance of applications that use large amounts of Sass stylesheets.

As to the overall build performance, can you also run the following commands (or the yarn variant if using it instead of npm):
npm ls @angular-devkit/build-angular
npm ls @ngtools/webpack
npm ls @angular/compiler-cli

[MathewBerg]

I'm running this on my computer which has just a slight different version, but I experience the same problem.

Here is the output:

`-- @angular-devkit/build-angular@0.1102.10

`-- @angular-devkit/build-angular@0.1102.10
  `-- @ngtools/webpack@11.2.10

`-- @angular/compiler-cli@11.2.11

We have roughly ~1700 components

[kemsky]

Sorry, does not seem to be related.

[MathewBerg]

Converting from scss to css resulted in a negligible speed difference.

[alan-agius4]

@MathewBerg, that is strange because based on the above timings the most of the time appears to be spent in sass-loader.

      {
        "averages": {
          "dataPoints": 1,
          "median": 202470,
          "mean": 202470,
          "range": {
            "start": 202470,
            "end": 202470
          }
        },
        "activeTime": 202470,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\mini-css-extract-plugin\\dist\\loader.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\css-loader\\dist\\cjs.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\postcss-loader\\src\\index.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\sass-loader\\dist\\cjs.js"
        ],
        "subLoadersTime": {}
      },
      {
        "averages": {
          "dataPoints": 1,
          "median": 194400,
          "mean": 194400,
          "range": {
            "start": 194400,
            "end": 194400
          }
        },
        "activeTime": 194400,
        "loaders": [
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\css-loader\\dist\\cjs.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\postcss-loader\\src\\index.js",
          "C:\\git\\bpi\\ng\\BPI.UI\\node_modules\\sass-loader\\dist\\cjs.js"
        ],
        "subLoadersTime": {}
      },

Is there a way to share the project privately or CPU profiles? Without neither it's hard to find the root cause.

[MathewBerg]

@alan-agius4 I have chrome-profiler-events.json, but it's about 600MB. Is there a preferred location you'd like me to upload it too?

[SirJosh1987]

I think I've figured this out

[alan-agius4]

@alan-agius4 I have chrome-profiler-events.json, but it's about 600MB. Is there a preferred location you'd like me to upload it too?

Not really as long as I can download them :)

[MathewBerg]

@alan-agius4 I added you to a private repo with the events in a .7z archive. I appreciate the help!

[alan-agius4]

@alan-agius4 I added you to a private repo with the events in a .7z archive. I appreciate the help!

Mind sharing the then uncompressed or in zip format? I don’t use 7zip.