Having many lazy chunks slows down rebuilds

[filipesilva]

Bug report

What is the current behavior?
Having many lazy chunks slows down rebuilds even when they don't change.

If the current behavior is a bug, please provide the steps to reproduce.
To repro follow the steps below. No tools aside from webpack and webpack-cli are used.

• git clone https://github.com/filipesilva/webpack-3500-lazy-chunks && cd webpack-3500-lazy-chunks
• npm install
• npm run watch and wait for the first build
• open ./out-tsc/app/app/app.module.ts
• add another console.log(1); to the end of the file and save
• the rebuild will take ~9000ms and emit only main.js and main.js.map
• progress report shows most of the time is spent in content hashing JavascriptModulesPlugin and after chunk asset optimization SourceMapDevtoolPlugin
• save the file without changes to trigger a similar long rebuild
• remove the import { AppRoutingModule } from './app-routing.module'; line (this file imports all the lazy modules) and save
• the rebuild now takes ~700ms

What is the expected behavior?
I think this is a bug because rebuild time should be mostly dominated by the size of changes to the affected chunks, and only minimally affected by other unchanged chunks.

Other relevant information:
webpack version: webpack@4.36.1
Node.js version: v10.16.0
Operating System: Windows 10 1903
Additional tools: webpack-cli@3.3.6

[filipesilva]

Also added a Chrome Devtools CPU profile to the repository now.

This is what the first rebuild, with console.log(1); added, looks like:

This is what the second rebuild, with just a save, looks like:

This is what the third rebuild, removing the lazy modules, looks like:

All three have a lot going on in seal. The first and second have a lot of time spent in lazyCompileHook. The third seems like a small version of the other two but is missing a lot of the createHash calls they have. I can't really interpret these very well but thought I'd put them up to help.

[sokra]

Seem to be caused by the ProgressPlugin. As workaround disable progress reporting.

[filipesilva]

I tried rebuilding by adding console.log(1); with variations of progress reporting and lazy chunks to see the difference:

• with lazy chunks, with --progress: ~8000ms
• with lazy chunks, without --progress: ~5000ms
• without lazy chunks, with --progress ~530ms
• without lazy chunks, without --progress ~480ms

It does seem like the progress reporting has an impact, but it's not obvious to me that it's the root cause. Taking progress reporting out seems to speed up things in general. With lazy chunks there is a lot more being written to the console so it would make sense that whatever extra computation is happening also causes more console writes, slowing it down further.

[filipesilva]

I've also now added a variation of the repo where all the imports are ESM static imports instead of dynamic ones. You can switch between them by commenting out one of the lines below, where app-routing-non-lazy.module contains the static imports:

import { AppRoutingModule } from './app-routing.module';
// import { AppRoutingModule } from './app-routing-non-lazy.module';

With this I tried rebuilding with and without both progress reporting (--progress) and sourcemaps (--devtool source-map):

• dynamic imports, with sourcemaps, with progress: ~8000ms
• dynamic imports, with sourcemaps, without progress: ~3000ms
• dynamic imports, without sourcemaps, without progress: ~2500ms
• static imports, with sourcemaps, with progress: ~2900ms
• static imports, with sourcemaps, without progress: ~2800ms
• static imports, without sourcemaps, without progress: ~600ms

I think turning off sourcemaps is an interesting comparison because I expected that changing a large chunk (because it imports all previously lazy chunks statically) with sourcemaps would be fairly expensive because the sourcemaps for the whole chunk would have to change. Indeed having the same large chunk but turning off sourcemaps resulted in 1/5 of the rebuild time. Even without sourcemaps I still expected a cost for rebuilding a new large string for that chunk though.

I expected that this problem could be mitigated by having dynamic imports splitting out the large chunk into many smaller chunks. When any single chunk changes, the sourcemap computation would be much smaller. But this didn't really happen, turning sourcemaps off with dynamic imports resulted only in a 5/6 rebuild time. I also noticed that rebuilding it without sourcemaps but with progress reporting will still spend a lot of time in content hashing JavascriptModulesPlugin.

So given the same total compilation size and shape I expected that splitting one large chunk into many small chunks would always result in smaller rebuild times in development mode. There might be cases where this shouldn't happen, but I think it would mostly be when the compilation moves imports around chunks.

[sokra]

Best use eval-source-map for the same quality, but better incremental build performance.

[sokra]

I fixed the ProgressPlugin performance in a PR, so progress should no longer make a bigger difference.

I figured out that most time is spend in Chunk Graph building. A lot of modules are added to all Chunks which are later removed by optimization. I will look into refactoring this and merging this optimization into the Graph building to avoid adding this unnecessary modules in the first step.

But after all this seem to be pretty fast. Your app has ~7,500 modules and 3,500 chunks and it rebuilds in 2.5s... Anyway the static imports case seem to suggest that <1s should be possible...

[filipesilva]

I do agree that 2.5s for 3,500 chunks is pretty fast, no real question about that. I don't know of any real project that has that many. I also agree that eval sourcemap is much better for these cases. At the limit if someone really wants speed no sourcemaps is the fastest option.

I got started looking at this because of angular/angular-cli#15086. I think what ends up happening in non-trivial setups and codebases is that there's a compounding effects.

A webpack setup has more plugins that might be hooking into those phases that get triggered for lazy chunks. In the Angular CLI setup I saw that happening with https://github.com/aackerman/circular-dependency-plugin I think.

Then even though no one has 3,500 lazy chunks, I have seen around ~120 chunk cases where some of those are much bigger, and some might be split chunks due to common modules. I didn't test those cases. But I expect that the lazy chunk computation that we see happening here is related to the number of modules in the lazy chunk, and maybe their relation to other chunks.

So I think in a real project that has less but bigger chunks and more plugin, the effect is more dramatic. There's also an argument that those computations would similarly affect the static import case. But I think that when that happens, splitting code into more chunks should speed up the rebuild instead of slowing it down.

Which is to say, I think that in real projects with a lot of code you will should faster rebuilds if you split into chunks using dynamic imports, instead of slower rebuilds.

[sokra]

#9472 should give some improvements

[sokra]

try 4.38.0

[filipesilva]

With 4.38.0 I see:

• dynamic imports, with sourcemaps, with progress: ~1600ms
• dynamic imports, with sourcemaps, without progress: ~1500ms
• dynamic imports, without sourcemaps, without progress: ~700ms
• static imports, with sourcemaps, with progress: ~3000ms
• static imports, with sourcemaps, without progress: ~3000ms
• static imports, without sourcemaps, without progress: ~600ms

So as far as I can tell, ProgressPlugin now doesn't really make a difference, sourcemap perf is better with dynamic chunks, and there doesn't seem to be much of a difference between dynamic imports vs static imports.

I think these results super impressive compared to the previous ones, between 2x and 5x faster rebuilds for the dynamic import scenario:

• dynamic imports, with sourcemaps, with progress: ~1600ms vs ~8000ms
• dynamic imports, with sourcemaps, without progress: ~1500ms vs ~3000ms
• dynamic imports, without sourcemaps, without progress: ~700ms vs ~2500ms

Awesome work @sokra!

As far as I'm concerned this particular issue is fixed, but if you feel like there's something else you'd like to look at feel free to reopen.

[sokra]

🎉