Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
ModuleConcatenationPlugin memory usage increases dramatically with chunk module count #5992
Do you want to request a feature or report a bug?
What is the current behavior?
If the current behavior is a bug, please provide the steps to reproduce.
This will result in peak memory usage of about 17GB.
The app has a 12 lazy loaded chunks by default.
Note: this repo uses a minimal
What is the expected behavior?
Please mention other relevant information such as the browser version, Node.js version, webpack version and Operating System.
referenced this issue
Nov 21, 2017
I took some CPU and heap profiles to see if they can shed some light on where memory is being used.
I'm controlling the number of modules by editing the lazy imports in
There are lazy 12 imports here, and each test was done by only having that number of imports uncommented (e.g.
CPU and heap profiles can be found in https://drive.google.com/drive/folders/1d4Cb-jB0gVADeIEtwcgt9tJOtS5dOOw-?usp=sharing. They can be opened by the Chrome profiler.
At this point I stopped because I don't think my computer could take the memory usage of both the webpack process and the chrome profiler together without having to aggressively garbage collect (which would mess with the profile numbers).
My lazy imports looked like this:
The trend seems solid though: each ~350 extra modules requires some 1.7g extra memory.
Worth noting that
Analyzing the CPU profiles shows growing time being used by the same functions:
Looking at the heap profiles it seems that the native
referenced this issue
Nov 22, 2017
Another interesting data point: trying to output webpack stats (
This seems to indicate something odd is happening to stats as a symptom of what's taking all the memory.
Perhaps the string is exceeding the maximum size allowed by node (https://github.com/nodejs/node/blob/master/deps/v8/include/v8.h#L2455, 32-bit environments is 256MB; 64-bit is 1GB for max string size as indicated by @clydin).
I tried with the smaller
A stats file for a compilation with one of the
I think I found something... following those repeated stats I tried removing the bit of code that adds the reasons in
And the memory usage collapsed completely. For the 7 imports case it went from 10.2GB down to 1.2GB.
I'm unsure of why this is, and why those reasons need to be added, but it's progress.
I think I found the problem.
The identifier for a concatenated module is the list of all file paths included in that module:
This string can get very, very big when there's a bunch of modules concatenated together. And everywhere that a module is used, if it's inside the concatenated module, it also uses the same identifier.
So if you had 350 modules their identifiers were, say, a path of 100 character length each. Now all those modules are inside a single concatenated module. The identifier of that concatenated module is 350 * 100 = 35,000 characters. That identifier is being used in at least 350 places, so at least 350 * 35,000 = 12,250,000 extra characters.
Changing this identifier to be a small string instead lowers the memory usage for the original repro from 17GB to 2.8GB.
Will put up a PR.
Special thanks to @clydin that pointed me at the increased size of the stats files and how it could be related to this problem.