New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModuleConcatenationPlugin memory usage increases dramatically with chunk module count #5992

Closed
filipesilva opened this Issue Nov 21, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@filipesilva
Contributor

filipesilva commented Nov 21, 2017

Do you want to request a feature or report a bug?
I want to report a bug.

What is the current behavior?
ModuleConcatenationPlugin memory usage increases dramatically with chunk module count.

If the current behavior is a bug, please provide the steps to reproduce.

git clone https://github.com/filipesilva/webpack-module-concatenation-memory
cd webpack-module-concatenation-memory
npm install
npm run ngc
npm run webpack

This will result in peak memory usage of about 17GB.

Commenting out new webpack.optimize.ModuleConcatenationPlugin(), in webpack.config.js results in
peak 1.8GB memory usage.

npm run ngc is necessary because it also generates some files in node_modules/ that are
imported by the app.

The app has a 12 lazy loaded chunks by default.
These can be imported into the main chunk by editing src/app/app-routing.module.js (after npm run ngc), commenting out the System.import: and uncommenting import declarations as shown in the repo readme. This will increase memory usage further.

Note: this repo uses a minimal webpack.config.json with no loaders:

const path = require('path');
const webpack = require('webpack');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const UglifyJSPlugin = require('uglifyjs-webpack-plugin');
const ProgressPlugin = require('webpack/lib/ProgressPlugin');

module.exports = {
  entry: './src/main.js',
  output: {
    path: path.resolve(__dirname, './dist'),
    filename: 'bundle.js'
  },
  plugins: [
    new ProgressPlugin(),
    // Ignore `Critical dependency: the request of a dependency is an expression` warnings.
    new webpack.ContextReplacementPlugin(
      /angular(\\|\/)core(\\|\/)esm5/,
      __dirname
    ),
    new HtmlWebpackPlugin({
      template: './src/index.html'
    }),
    new webpack.optimize.ModuleConcatenationPlugin(),
  ]
};

What is the expected behavior?
Memory usage should be more in line with base webpack usage (around 1.8GB instead of 17GB).

Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.
No browser was used, node v8.9.1, webpack 3.8.1, Windows 10

@sokra sokra added this to the webpack 4 milestone Nov 21, 2017

@filipesilva

This comment has been minimized.

Contributor

filipesilva commented Nov 21, 2017

@filipesilva

This comment has been minimized.

Contributor

filipesilva commented Nov 22, 2017

I took some CPU and heap profiles to see if they can shed some light on where memory is being used.

I'm controlling the number of modules by editing the lazy imports in ./src/app/app-routing.module.js lines 52-63:

System.import('./cg/cg.module.ngfactory.js');
System.import('./mod1/mod1.module.ngfactory.js');
System.import('./mod2/mod2.module.ngfactory.js');
System.import('./mod3/mod3.module.ngfactory.js');
System.import('./mod4/mod4.module.ngfactory.js');
System.import('./mod5/mod5.module.ngfactory.js');
System.import('./mod6/mod6.module.ngfactory.js');
System.import('./mod7/mod7.module.ngfactory.js');
System.import('./mod8/mod8.module.ngfactory.js');
System.import('./mod9/mod9.module.ngfactory.js');
System.import('./mod10/mod10.module.ngfactory.js');
System.import('./root/root.module.ngfactory.js');

There are lazy 12 imports here, and each test was done by only having that number of imports uncommented (e.g. 3 imports is the first three). Peak memory was measured by looking at the task manager.

1 import:

  • peak memory used: ~260mb
  • number of modules: ~700

2 imports:

  • peak memory used: ~1.6gb
  • number of modules: ~1200

3 imports:

  • peak memory used: ~2.4gb
  • number of modules: ~1600

4 imports:

  • peak memory used: ~5.2gb
  • number of modules: ~1900

5 imports:

  • peak memory used: ~6.7gb
  • number of modules: ~2200

6 imports:

  • peak memory used: ~8.7gb
  • number of modules: ~2600

7 imports:

  • peak memory used: ~10.2gb
  • number of modules: ~2900

CPU and heap profiles can be found in https://drive.google.com/drive/folders/1d4Cb-jB0gVADeIEtwcgt9tJOtS5dOOw-?usp=sharing. They can be opened by the Chrome profiler.

At this point I stopped because I don't think my computer could take the memory usage of both the webpack process and the chrome profiler together without having to aggressively garbage collect (which would mess with the profile numbers).

My lazy imports looked like this:

System.import('./cg/cg.module.ngfactory.js');
System.import('./mod1/mod1.module.ngfactory.js');
System.import('./mod2/mod2.module.ngfactory.js');
System.import('./mod3/mod3.module.ngfactory.js');
System.import('./mod4/mod4.module.ngfactory.js');
System.import('./mod5/mod5.module.ngfactory.js');
System.import('./mod6/mod6.module.ngfactory.js');
// System.import('./mod7/mod7.module.ngfactory.js');
// System.import('./mod8/mod8.module.ngfactory.js');
// System.import('./mod9/mod9.module.ngfactory.js');
// System.import('./mod10/mod10.module.ngfactory.js');
// System.import('./root/root.module.ngfactory.js');

The trend seems solid though: each ~350 extra modules requires some 1.7g extra memory.

Worth noting that 1-import is a bit of an odd case because it's a smaller lazy module (/cg/cg.module.ngfactory.js). The modX/modX.module.ngfactory.js ones are all the same size (350ish modules).

Analyzing the CPU profiles shows growing time being used by the same functions:

2-import:
image

7-import:
image

dep.module.reasons.forEach.reason and _orderedConcatenationList.map.info are both in ./lib/optimize/ModuleConcatenationPlugin.js and see growing time spent. DoJoin and writeUtf8String are native functions but also seem to be used more and more.

Looking at the heap profiles it seems that the native stringSlice is what's eating up all the memory, followed by DoJoin:

2-import:
image

7-import:
image

@filipesilva

This comment has been minimized.

Contributor

filipesilva commented Nov 22, 2017

Another interesting data point: trying to output webpack stats (npm run webpack -- --json > stats.json) will fail with any of the lazy loaded modX/modX.module.ngfactory.js modules:

D:\sandbox\webpack-module-concatenation-memory\node_modules\webpack\bin\webpack.js:367
                                process.stdout.write(JSON.stringify(stats.toJson(outputOptions), null, 2) + "\n");
                                                          ^

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at compilerCallback (D:\sandbox\webpack-module-concatenation-memory\node_modules\webpack\bin\webpack.js:367:31)

This seems to indicate something odd is happening to stats as a symptom of what's taking all the memory.

Perhaps the string is exceeding the maximum size allowed by node (https://github.com/nodejs/node/blob/master/deps/v8/include/v8.h#L2455, 32-bit environments is 256MB; 64-bit is 1GB for max string size as indicated by @clydin).

I tried with the smaller ./cg/cg.module.ngfactory.js and was able to output a 25MB stats file. Without ModuleConcatenationPlugin the stats file is 16MB.

A stats file for a compilation with one of the modX/modX.module.ngfactory.js modules also without ModuleConcatenationPlugin is 26MB.

Edit:
There also seems to be a bunch of repeats in the chunk module reasons stats:
image

@filipesilva

This comment has been minimized.

Contributor

filipesilva commented Nov 22, 2017

I think I found something... following those repeated stats I tried removing the bit of code that adds the reasons in ModuleConcatenationPlugin:

newModule.dependencies.forEach(dep => {
if(dep.module) {
dep.module.reasons.forEach(reason => {
if(reason.dependency === dep)
reason.module = newModule;
});
}
});

And the memory usage collapsed completely. For the 7 imports case it went from 10.2GB down to 1.2GB.

I'm unsure of why this is, and why those reasons need to be added, but it's progress.

@filipesilva

This comment has been minimized.

Contributor

filipesilva commented Nov 22, 2017

I think I found the problem.

The identifier for a concatenated module is the list of all file paths included in that module:

identifier() {
return this._orderedConcatenationList.map(info => {
switch(info.type) {
case "concatenated":
return info.module.identifier();
}
}).filter(Boolean).join(" ");
}

This string can get very, very big when there's a bunch of modules concatenated together. And everywhere that a module is used, if it's inside the concatenated module, it also uses the same identifier.

So if you had 350 modules their identifiers were, say, a path of 100 character length each. Now all those modules are inside a single concatenated module. The identifier of that concatenated module is 350 * 100 = 35,000 characters. That identifier is being used in at least 350 places, so at least 350 * 35,000 = 12,250,000 extra characters.

Changing this identifier to be a small string instead lowers the memory usage for the original repro from 17GB to 2.8GB.

Will put up a PR.

Special thanks to @clydin that pointed me at the increased size of the stats files and how it could be related to this problem.

filipesilva added a commit to filipesilva/webpack that referenced this issue Nov 22, 2017

filipesilva added a commit to filipesilva/webpack that referenced this issue Nov 22, 2017

@sokra sokra closed this in #5997 Nov 22, 2017

elliottsj added a commit to elliottsj/web-build-tools that referenced this issue Apr 4, 2018

Upgrade webpack to ~3.11.0
webpack@~3.11.0 includes a fix for an out-of-memory issue; see:
webpack/webpack#5992
webpack/webpack#5997

elliottsj added a commit to elliottsj/web-build-tools that referenced this issue Apr 4, 2018

Upgrade webpack to ~3.11.0
webpack@~3.11.0 includes a fix for an out-of-memory issue; see:
webpack/webpack#5992
webpack/webpack#5997

nickpape-msft added a commit to Microsoft/web-build-tools that referenced this issue Apr 5, 2018

Upgrade webpack to ~3.11.0 (#600)
* Upgrade webpack to ~3.11.0

webpack@~3.11.0 includes a fix for an out-of-memory issue; see:
webpack/webpack#5992
webpack/webpack#5997

* rush change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment