[webpack 5] rebuild is twice as slow comparing to webpack 4 #12102

meskill · 2020-12-03T07:35:00Z

After upgrading to webpack 5 in our project we noticed that rebuild time increased more than 2 times comparing to webpack 4.

I've created a test repo https://github.com/meskill/webpack5-performance-issue to show the difference in rebuild time for webpack4 and webpack5.

It shows more than 2-time slowness of rebuild in webpack5 compare to webpack4. Profiling shows similar info

Cpuprofile for both webpack4 and webpack5

alexander-akait · 2020-12-03T11:21:40Z

Thank you for creating this issue. However, issues need to follow one of our templates so that we can clearly understand your particular circumstances.

Please help us help you by recreating the issue using one of our templates.

Please respect time of other developers

Thebarda · 2020-12-03T14:38:23Z

Hello, I have the same issue here. I have compared build time with Webpack 4 and Webpack 5 on a project using Speed Measure Plugin and here is the result :

As you can see, the time took by plugins ands loaders is about the same but Webpack 5 is twice slower than Webpack 4 is my case.
I can provide more informations if needed.

sokra · 2020-12-03T14:43:42Z

@Thebarda please attach cpu profiles

--progress profile also displays some timing information.

alexander-akait · 2020-12-03T14:52:18Z

I think bug in HtmlWebpackHarddiskPlugin

Thebarda · 2020-12-03T15:14:06Z

Here is the cpuprofile for a Webpack 4 build

And the cpuprofile for Webpack 5 build.

Indeed @alexander-akait, for the webpack 5 build I'm using html-webpack-plugin on next branch (html-webpack-plugin@5 alpha) that supports Webpack 5 and html-webpack-harddisk-plugin@1.0.2

alexander-akait · 2020-12-03T15:17:16Z

What is version of terser-webpack-plugin, please provide configuration, it is really hard each time to ask you about versions, options and etc

alexander-akait · 2020-12-03T15:17:52Z

Why you run rebuild in production mode?

alexander-akait · 2020-12-03T15:19:20Z

What is node version? Node@10 has regression in perf

sokra · 2020-12-04T07:57:45Z

@Thebarda Seems like you are talking about build performance and not rebuild performance. This issue is about rebuild performance, so seem like your issue is not similar to @meskill's issue, and I beg you to open an separate issue for your unrelated issue. It looks strongly like you are using persistent cache for terser-webpack-plugin in webpack 4 and measuring the cached build. While in webpack 5 you have to opt-in into persistent caching. Please make sure to remove node_modules/.cache before measuring.

sokra · 2020-12-04T09:11:06Z

@meskill i looked at your profile and noticed a few things:

version	change to update time	compile time
webpack 4	~850ms	~500ms
webpack 5	~1300ms	~950ms

So we are talking about 450ms extra for webpack 5.

module graph validation takes about ~100ms extra in webpack 5
- webpack 5 is using a safe cache. webpack 4 uses an unsafe cache
- You can change that with module.unsafeCache: true, but I don't recommend that
ModuleInfoHeaderPlugin costs about ~100ms extra in webpack 5
- It prints more info about exports, etc.
- You can disable that with output.pathinfo: false
- We should add caching to this plugin
- Do we really need that detailed information by default? Maybe boolean | "verbose" and default to simple information.
Stats.toString takes about ~140ms extra in webpack 5
- Seems like webpack 4 skips cached modules early, while webpack 5 extracts data for them before skiping them later
- You can disable modules with stats.modules: false, but this would hide all modules
- We should not extra data for hidden modules during stats generation
In both versions the change is pushed pretty late to the web socket
- watchers are applied before that. They cost about ~50ms (webpack 5) resp. ~15ms (webpack 4)
- node-notifier is triggered in webpack 4. That costs about ~35ms
- I wonder if we should add a delay to the watchers?
Both versions wait 300ms after a file change before starting the build
- This happens to merges multiple files changes that happen in fast sequence (like batch replace or whatever)
- You could change that with watchOptions.aggregateTimeout: 50. Not sure if this worth it.

meskill · 2020-12-04T09:19:59Z

@sokra Thanks, for the suggestions, I will check it.

Also, I've changed the initial issue description with some simple repo to neglect our project-specific config - in this repo webpack4 and webpack5 use the same config for the react example project.

P.S.: yes we already fixed the problem with long stats.toJSON by setting all: false and enabling only needed information about the build what made stats.JSON pretty fast now

keepitterron · 2020-12-06T22:34:23Z

This is compatible with what I'm seeing in a large, multi entries app when I upgrade. Building is a bit slower (~18s to ~22s), rebuilding in watch mode is up to 3x slower.

Development mode

  optimization: {
    moduleIds: 'hashed',
    splitChunks: {
      cacheGroups: {
        default: false,
        commons: {
          name: 'commons',
          chunks: 'all',
          minChunks: 3,
        },
      },
    },
  },

webpack 4

[webpack-cli] Compilation starting...
<i> [webpack.Progress] 216ms building
<i> [webpack.Progress] 12ms finish module graph
<i> [webpack.Progress] 155ms chunk graph
<i> [webpack.Progress] 183ms advanced chunk optimization
<i> [webpack.Progress] 84ms before module ids
<i> [webpack.Progress] 23ms module id optimization
<i> [webpack.Progress] 142ms hashing
<i> [webpack.Progress] 12ms additional chunk assets processing
<i> [webpack.Progress] 41ms emitting
[webpack-cli] Compilation finished
Hash: ac5359db523b6792fc80
Version: webpack 4.44.1
Time: 917ms

webpack 5

[webpack-cli] Compilation starting...
<i> [webpack.Progress]  | 20 ms setup > compile
<i> [webpack.Progress]  |  | 14 ms compilation > AssetModulesPlugin
<i> [webpack.Progress]  | 27 ms setup > compilation
<i> [webpack.Progress] 50 ms setup
<i> [webpack.Progress]  | 608 ms building > entries dependencies modules
<i> [webpack.Progress]  | 506 ms building > entries dependencies modules
<w> [webpack.Progress] 1116 ms building
<i> [webpack.Progress]  |  | 85 ms finish module graph > FlagDependencyExportsPlugin
<i> [webpack.Progress]  | 147 ms sealing > finish module graph
<i> [webpack.Progress]  | 181 ms sealing > chunk graph
<i> [webpack.Progress]  |  | 224 ms chunk optimization > SplitChunksPlugin
<i> [webpack.Progress]  | 232 ms sealing > chunk optimization
<i> [webpack.Progress]  |  | 34 ms module ids > HashedModuleIdsPlugin
<i> [webpack.Progress]  | 35 ms sealing > module ids
<i> [webpack.Progress]  | 303 ms sealing > module hashing
<i> [webpack.Progress]  | 35 ms sealing > code generation
<i> [webpack.Progress]  | 31 ms sealing > runtime requirements
<i> [webpack.Progress]  | 52 ms sealing > hashing
<i> [webpack.Progress]  |  | 22 ms asset processing > copy-webpack-plugin
<i> [webpack.Progress]  | 22 ms sealing > asset processing
<i> [webpack.Progress]  | 11 ms sealing > after asset optimization
<w> [webpack.Progress] 1079 ms sealing
<i> [webpack.Progress]  | 38 ms emitting > emit
<i> [webpack.Progress] 38 ms emitting
99% done plugins webpack-cli[webpack-cli] Compilation finished
assets by status 86.9 MiB [cached] 352 assets
cached modules 24.1 MiB (javascript) 147 KiB (runtime) [cached] 5480 modules
javascript modules 2.79 KiB
  ./assets/img/logos/ sync ^\.\/.*$ 870 bytes [built]
  ./src/client/js/components/ui-components/emoji/index.js 1.43 KiB [built]
  ./assets/img/video-player/ sync ^\.\/.*.*\.png$ 530 bytes [built]

webpack 5.10.0 compiled with 1 warning in 2283 ms

alexander-akait · 2020-12-07T10:41:53Z

Why you use splitChunks in development mode?

keepitterron · 2020-12-07T11:40:57Z

@alexander-akait I see non-trivial improvements in CPU and memory usage using split chunks in development with webpack4.

Anyways, taking optimization out in dev mode changes nothing (I rm -rf node_modules for good measure before running the test)

[webpack-cli] Compilation starting...
<i> [webpack.Progress]  | 23 ms setup > compile
<i> [webpack.Progress]  |  | 12 ms compilation > EntryPlugin
<i> [webpack.Progress]  | 15 ms setup > compilation
<i> [webpack.Progress] 38 ms setup
<i> [webpack.Progress]  | 763 ms building > entries dependencies modules
<i> [webpack.Progress]  | 34 ms building > entries dependencies modules
<i> [webpack.Progress] 797 ms building
<i> [webpack.Progress]  |  | 174 ms finish module graph > FlagDependencyExportsPlugin
<i> [webpack.Progress]  | 271 ms sealing > finish module graph
<i> [webpack.Progress]  |  | 38 ms plugins > WarnCaseSensitiveModulesPlugin
<i> [webpack.Progress]  | 39 ms sealing > plugins
<i> [webpack.Progress]  | 293 ms sealing > chunk graph
<i> [webpack.Progress]  |  | 12 ms chunk optimization > EnsureChunkConditionsPlugin
<i> [webpack.Progress]  |  | 343 ms chunk optimization > SplitChunksPlugin
<i> [webpack.Progress]  | 356 ms sealing > chunk optimization
<i> [webpack.Progress]  |  | 25 ms module ids > NamedModuleIdsPlugin
<i> [webpack.Progress]  | 26 ms sealing > module ids
<i> [webpack.Progress]  | 625 ms sealing > module hashing
<i> [webpack.Progress]  | 58 ms sealing > code generation
<i> [webpack.Progress]  | 104 ms sealing > runtime requirements
<i> [webpack.Progress]  | 90 ms sealing > hashing
<i> [webpack.Progress]  |  | 12 ms asset processing > copy-webpack-plugin
<i> [webpack.Progress]  | 12 ms sealing > asset processing
<i> [webpack.Progress]  | 18 ms sealing > after asset optimization
<w> [webpack.Progress] 1922 ms sealing
<i> [webpack.Progress]  | 29 ms emitting > emit
<i> [webpack.Progress] 29 ms emitting
99% done plugins webpack-cli[webpack-cli] Compilation finished
assets by status 622 MiB [cached] 354 assets
cached modules 24.1 MiB (javascript) 57.6 KiB (runtime) [cached] 5415 modules
./src/client/js/components/ui-components/emoji/index.js 1.43 KiB [built]
webpack 5.10.0 compiled successfully in 2787 ms

I'm really not sure which way to go. Webpack 4 is MUCH faster on rebuilds.
I will try to put together a reproducible repo but it's gonna be hard, this is a big private app.

keepitterron · 2020-12-07T12:09:50Z

On splitChunks in dev mode I see other people ran into the same issue and used the same solution:
#8644 (comment)

My reason was that when you have chunking-by-default enabled, a change of a single source file causes a single, small chunk+sourcemap to be re-rendered, emitting e.g. ~100KB of code.

On the other hand, when you have a single chunk which is e.g. 50MB and you always have to emit it (and its sourcemap, which is especially expensive). This also was problematic for browsers, as they would choke while loading huge sourcemaps like that.

joshxyzhimself · 2020-12-11T05:20:11Z

Aside from worse compile times, has anyone noticed the larger memory build-up?

On subsequent edits and re-bundling of my code my ram usage just keeps stacking up.

alexander-akait · 2020-12-11T12:06:22Z

@joeldenning with webpack-dev-server or simple bundle?

jolyndenning · 2020-12-11T18:28:54Z

@joeldenning with webpack-dev-server or simple bundle?

I think you may have tagged me instead of @joshxyzhimself 😄

joshxyzhimself · 2020-12-12T10:12:40Z

😸 😸 😸

@alexander-akait just simple bundle sir

edit file -> recompile -> ram usage increase
edit file again -> recompile -> ram usage increase
edit file again -> recompile -> ram usage increase

current workaround i do is kill the process and rerun it

i was thinking of writing a plugin that exposes and calls global.gc() but there may be other ways i'm not aware of? 😸

const fs = require('fs');
const os = require('os');
const path = require('path');
const assert = require('assert');
const process = require('process');
const webpack = require('webpack');
const TerserPlugin = require('terser-webpack-plugin');

const process_cwd = process.cwd();

const chunks_path = path.join(process_cwd, '.chunks.json');
const output_path = path.join(process_cwd, '/client-dist');

class chunk_extract_plugin {
  constructor(mode) {
    assert(typeof mode === 'string');
    this.mode = mode;
  }
  apply(compiler) {
    compiler.hooks.done.tap('Hello World Plugin', (stats) => {

      const chunks_set = new Set();

      if (this.mode === 'production') {
        stats.compilation.chunks.forEach((chunk) => {
          chunk.files.forEach((file) => {
            if (path.extname(file) === '.js') {
              chunks_set.add(file);
            }
          });
        });
      }

      stats.compilation.entrypoints.forEach((entrypoint) => {
        entrypoint.chunks.forEach((chunk) => {
          chunk.files.forEach((file) => {
            if (path.extname(file) === '.js') {
              chunks_set.add(file);
            }
          });
        });
      });

      const chunks_array = Array.from(chunks_set);

      fs.writeFileSync(chunks_path, JSON.stringify(chunks_array, null, 2));
    });
  }
}

const Client = (env, argv) => {
  const { mode } = argv;
  return {
    devtool: mode === 'development' ? 'source-map' : false,
    entry: ['./client/client.jsx'],
    resolve: {
      extensions: ['.js', '.jsx'],
    },
    module: {
      rules: [
        {
          enforce: 'pre',
          test: /\.(js|jsx)$/,
          exclude: /node_modules/,
          use: [
            {
              loader: 'eslint-loader',
              options: {
                fix: true,
                cache: true,
                emitWarning: true,
              },
            },
          ],
        },
        {
          test: /\.(js|jsx)$/,
          use: 'babel-loader',
          exclude: /node_modules/,
        },
        {
          test: /\.(css)$/,
          use: [
            { loader: 'style-loader' },
            { loader: 'css-loader', options: { importLoaders: 1 } },
            { loader: 'postcss-loader' },
          ],
        },
        {
          test: /\.(woff(2)?|ttf|eot|otf|svg)(\?v=\d+\.\d+\.\d+)?$/,
          use: [
            {
              loader: 'file-loader',
              options: {
                name: '[name].[ext]',
                outputPath: 'fonts/',
              },
            },
          ],
        },
      ],
    },
    plugins: [
      new webpack.DefinePlugin({ ENVIRONMENT: JSON.stringify(mode) }),
      new chunk_extract_plugin(mode),
    ],
    optimization: {
      runtimeChunk: true,
      splitChunks: { chunks: 'all' },
      minimize: mode === 'production',
      minimizer: [
        new TerserPlugin({
          parallel: os.cpus().length,
          terserOptions: {
            output: { comments: false },
            compress: { dead_code: true },
            mangle: true,
          },
        }),
      ],
    },
    output: {
      filename: mode === 'production' ? '[chunkhash].js' : '[name].js',
      chunkFilename: mode === 'production' ? '[chunkhash].js' : '[name].js',
      path: output_path,
      publicPath: '/scripts/',
    },
    stats: 'minimal',
  };
};

module.exports = [Client];

joshxyzhimself · 2020-12-12T13:13:02Z

is webpack the new slack? 😸

sokra · 2020-12-12T19:00:17Z

@joshxyzhimself if you create a new issue with a repro repo, we can help to find the memory leak. Please create a new issue as this is unreleated to the rebuild performance issue.

meskill · 2020-12-16T07:06:24Z

@sokra hello, I've tested some of the settings you suggested and it improved time a little bit, but webpack4 still beats webpack5 regarding rebuild time. changes

Does webpack team have any plans to improve performance for the rebuild in near future?

alexander-akait · 2020-12-16T12:06:59Z

@meskill do you update webpack to the latest 5 version? Maybe you can provide profile file?

alexander-akait · 2021-01-18T18:04:50Z

@meskill can you try latest webpack version and put feedback more here?

alexander-akait · 2021-01-18T18:08:10Z

@keepitterron Can you do the same and put feedback here, thanks

alexander-akait · 2021-12-30T12:48:11Z

defaultWebpack: ['webpack/lib/'] why you use it? webpack invalidate caches after update version, you should not do it. Can you try to disable cacheUnaffected? backCompat can be disable if you use plugins/loaders for webpack v5.

But HRM looks for a lot of time in module building

How many?

xyxiao001 · 2021-12-30T12:52:47Z

I tried to disable cacheUnaffected backCompat，waiting for me to try

xyxiao001 · 2021-12-30T13:06:54Z

webpack 5.65.0
config

cache: isProd ? false : {
      name: `${createHash(includeChunks)}-${includeChunks}`,
      type: 'filesystem',
 },

config.experiments = {
      lazyCompilation: {
         entries: true,
          imports: true,
      },
      futureDefaults: true,
      outputModule: true,
      css: true,
 };
 config.module.unsafeCache = true;
 config.output.pathinfo = false;
 config.stats.all = false;
 config.optimization.splitChunks = false;
 config.optimization.sideEffects = false;
 config.optimization.providedExports = false;

devserve

DONE  Compiled successfully in 462273ms                                                                            
 SMP  ⏱  
General output time took 7 mins, 42.3 secs
 SMP  ⏱  Loaders
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
@vue/cli-service took 4 mins, 4.78 secs
  module count = 1769
babel-loader, and 
cache-loader, and 
@vue/cli-service took 3 mins, 10.77 secs
  module count = 1711
@vue/vue-loader-v15, and 
cache-loader, and 
@vue/cli-service took 3 mins, 2.9 secs
  module count = 3388
cache-loader, and 
@vue/cli-service took 2 mins, 16.96 secs
  module count = 1711
babel-loader took 1 min, 39.004 secs
  module count = 686
modules with no loaders took 1 min, 30.28 secs
  module count = 3075
svg-sprite-loader, and 
svgo-loader took 1 min, 25.69 secs
  module count = 172
css-loader, and 
postcss-loader, and 
sass-loader took 1 min, 3.94 secs
  module count = 8
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
cache-loader, and 
@vue/cli-service took 48.27 secs
  module count = 12
svgo-loader took 44.67 secs
  module count = 220
css-loader, and 
postcss-loader took 19.99 secs
  module count = 30
@vue/vue-loader-v15, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
@vue/cli-service took 3.45 secs
  module count = 3538
cache-loader, and 
font-cloak took 2.095 secs
  module count = 10
@vue/vue-loader-v15, and 
babel-loader, and 
cache-loader, and 
@vue/cli-service took 0.795 secs
  module count = 1711
vue-style-loader, and 
css-loader, and 
postcss-loader took 0.023 secs
  module count = 30
html-webpack-plugin took 0.016 secs
  module count = 1
@vue/vue-loader-v15, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
cache-loader, and 
@vue/cli-service took 0.012 secs
  module count = 24
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader took 0.009 secs
  module count = 8

hrm

 DONE  Compiled successfully in 6785ms                                                                                   
 SMP  ⏱  
General output time took 6.78 secs

 SMP  ⏱  Loaders
cache-loader, and 
@vue/cli-service took 1.29 secs
  module count = 1
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
@vue/cli-service took 0.434 secs
  module count = 1
css-loader, and 
@vue/vue-loader-v15, and 
postcss-loader, and 
cache-loader, and 
@vue/cli-service took 0.388 secs
  module count = 1
cache-loader, and 
@vue/vue-loader-v15, and 
cache-loader, and 
@vue/cli-service took 0.379 secs
  module count = 1
babel-loader, and 
cache-loader, and 
@vue/cli-service took 0.366 secs
  module count = 1

webpack 4.42.1
devserve

 DONE  Compiled successfully in 326894ms                                                                                                            下午9:05:35



 SMP  ⏱  
General output time took 5 mins, 26.91 secs
 SMP  ⏱  Plugins
ModuleFederationPlugin took 5 mins, 10.84 secs
 SMP  ⏱  Loaders
css-loader, and 
vue-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 5 mins, 4.26 secs
  module count = 1771
cache-loader, and 
vue-loader took 4 mins, 48.66 secs
  module count = 5108
vue-loader, and 
cache-loader, and 
babel-loader, and 
cache-loader, and 
vue-loader took 4 mins, 37.51 secs
  module count = 3428
cache-loader, and 
babel-loader took 2 mins, 28.69 secs
  module count = 696
svg-sprite-loader, and 
svgo-loader took 2 mins, 17.41 secs
  module count = 172
css-loader, and 
postcss-loader, and 
sass-loader took 1 min, 52.3 secs
  module count = 8
modules with no loaders took 1 min, 50.96 secs
  module count = 2338
css-loader, and 
vue-loader, and 
postcss-loader, and 
cache-loader, and 
vue-loader took 50.4 secs
  module count = 12
file-loader, and 
svgo-loader took 46.29 secs
  module count = 220
url-loader took 26.075 secs
  module count = 723
css-loader, and 
postcss-loader took 24.83 secs
  module count = 28
vue-loader, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 3.013 secs
  module count = 3542
url-loader, and 
cache-loader, and 
font-cloak took 1.38 secs
  module count = 10
raw-loader took 0.652 secs
  module count = 5
html-webpack-plugin took 0.06 secs
  module count = 1
vue-style-loader, and 
css-loader, and 
postcss-loader took 0.053 secs
  module count = 28
vue-loader, and 
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
cache-loader, and 
vue-loader took 0.018 secs
  module count = 24
vue-style-loader, and 
css-loader, and 
postcss-loader, and 
sass-loader took 0.013 secs
  module count = 8

hrm

 DONE  Compiled successfully in 4424ms                                                                                                              下午9:06:30



 SMP  ⏱  
General output time took 4.42 secs
 SMP  ⏱  Plugins
ModuleFederationPlugin took 0.198 secs
 SMP  ⏱  Loaders
cache-loader, and 
vue-loader took 0.491 secs
  module count = 2
css-loader, and 
vue-loader, and 
postcss-loader, and 
sass-loader, and 
cache-loader, and 
vue-loader took 0.312 secs
  module count = 2
cache-loader, and 
babel-loader, and 
cache-loader, and 
vue-loader took 0.201 secs
  module count = 1

After the upgrade, HRM spends more time in building module

alexander-akait · 2021-12-30T13:15:56Z

I don't know what is @vue/cli-service, but it is very slow on initial build, very... Also svgo-loader eats very many time (do you need it in develoment mode ? I think it is useful only for production). Also check the mode option? Do you use development for development, because production requires more time?

alexander-akait · 2021-12-30T13:18:36Z

Also name: ${createHash(includeChunks)}-${includeChunks}, is the same during multiple runs without modification configuration files? How you calculate it?

xyxiao001 · 2021-12-30T13:19:38Z

@vue/cli-service is the cli provided by Vue, encapsulated some configuration items.

xyxiao001 · 2021-12-30T13:20:20Z

Also name: ${createHash(includeChunks)}-${includeChunks}, is the same during multiple runs without modification configuration files? How you calculate it?

Cache by the imported module name, can start a module separately

xyxiao001 · 2021-12-30T13:21:27Z

Thank you for answering, I am trying to remove the vue-cli, use webpack5 directly,

alexander-akait · 2021-12-30T13:34:34Z

Cache by the imported module name, can start a module separately

It should be deteriorated between different builds, i.e. if you run webpack and then stop and then run again, the value should be same

VishalBhandare-Apertumo · 2022-04-10T17:28:58Z

no recent communication on this issue.
is this fixed or will never be solved?
can someone suggest alternative for problem?
few configuration changes are not making webpack5 build faster than webpack4.

alexander-akait · 2022-04-11T11:33:58Z

@VishalBhandare-Apertumo Please provide profile as minimum (examples are above)

VishalBhandare-Apertumo · 2022-04-11T11:52:02Z

We downgraded to webpack4 as seems there was this issue with webpack5. i have moved to new work now. But yes i need to find solution for this problem soon or later. i will communicating in this thread when i start working on fixing this.

@alexander-akait you have been very actively helping people here. thanks a lot.

I will get back to you while debugging issue if required.

For Vue Cli users linking references
https://github.com/vuejs/vue-cli/issues/6212

alexander-akait · 2022-04-11T11:58:08Z

Yes, will be great to profile and see there is perf problem (maybe in some loaders/plugins?) Feel free to ping me when you will do it

FishOrBear · 2022-04-21T03:47:54Z

Does ts-loader not support cache?
If I remove the cache-loader, the build time becomes 30s, if I add the cache-laoder, it will be within 10s

   "webpack": "^5.72.0",
        "webpack-cli": "^4.9.2",
        "webpack-dev-server": "^4.8.1",
        "webpack-merge": "^5.8.0",
        "webpackbar": "^5.0.2",

        "ts-loader": "^9.2.8",

vankop · 2022-04-21T08:55:44Z

cache-loader is deprecated, you don't need it with webpack@5

noppa · 2022-04-21T10:19:13Z

@FishOrBear Have you tried with Webpack's own filesystem cache? It's off (or rather in-memory only) by default.

  cache: {
    type: 'filesystem',
    compression: 'gzip',
  },

should speed up subsequent starts considerably

FishOrBear · 2022-04-22T01:27:47Z

I have started in webpack.common.ts

const config: webpack.Configuration = {
     cache: {
         type: 'filesystem',
     },

FishOrBear · 2022-04-22T01:30:22Z

Thanks for your reply, I know it's been stopped for maintenance, but if I don't use it(cache-loader), it's slow.

sibelius · 2022-07-12T15:39:40Z

lazy compilation makes compilation faster

should we close this issue ?

and open new issues more specific ?

alexander-akait · 2022-07-12T20:43:57Z

@sibelius this issue for tracking perf problems, because in some places we can have potentials improvements or bugs/memory leaks (but yes lazy compilation do your compiler faster)

meskill · 2022-12-05T10:36:32Z

We noticed the big build/rebuild slowdowns on the big applications with a lot of dynamic imports.
The most noticeable time is spent in visitModules block (more than 4s).

Webpack logs with dynamic imports

100% (31840 affected + 0 infected of 31840) modules flagged as affected (31840 new modules, 0 changed, 0 references changed, 0 unchanged, 0 were not built) compute affected modules: 445.484298 ms finish modules: 19.2084 ms report dependency errors and warnings: 427.454098 ms optimize dependencies: 0.3432 ms create chunks: 4727.013482 ms 0% modules flagged as affected by chunk graph (31840 new modules, 0 changed, 0 unchanged) compute affected modules with chunk graph: 217.759399 ms optimize: 701.219197 ms 27958 modules hashed, 0 from cache (0.88 variants per module in average) module hashing: 1225.826896 ms 100% code generated (27956 generated, 0 from cache) code generation: 2767.025084 ms runtime requirements.modules: 229.206198 ms runtime requirements.chunks: 709.575096 ms runtime requirements.entries: 101.976499 ms runtime requirements: 1044.298593 ms hashing: initialize hash: 0.0097 ms hashing: sort chunks: 0.8831 ms hashing: hash runtime modules: 7641.072852 ms hashing: hash chunks: 2737.142484 ms hashing: hash digest: 1.9061 ms hashing: process full hash modules: 0.0078 ms hashing: 10394.245136 ms 100% code generated (23 generated, 0 from cache) record hash: 0.1943 ms module assets: 23.545 ms create chunk assets: 23352.211563 ms process assets: 13429.300045 ms

LOG from webpack.Compiler
make hook: 515618.904476 ms
finish make hook: 320.076499 ms
finish compilation: 894.234496 ms
seal compilation: 58097.3293 ms
afterCompile hook: 1.5714 ms
emitAssets: 37881.879832 ms
emitRecords: 2.1531 ms

LOG from webpack.ResolverCachePlugin
29% really resolved (46940 real resolves with 0 cached but invalid, 113980 cached valid, 0 concurrent)

LOG from webpack.buildChunkGraph
visitModules: prepare: 43.4525 ms
visitModules: visiting: 118.5766 ms
visitModules: calculating available modules: 1.5126 ms
visitModules: merging available modules: 0.5015 ms
visitModules: check modules for revisit: 0.2055 ms
visitModules: prepare: 43.6924 ms
visitModules: visiting: 172.638099 ms
visitModules: calculating available modules: 0.4296 ms
visitModules: merging available modules: 0.7218 ms
visitModules: check modules for revisit: 0.1626 ms
visitModules: prepare: 98.670599 ms
visitModules: visiting: 4121.603685 ms
visitModules: calculating available modules: 129.659099 ms
visitModules: merging available modules: 90.8755 ms
visitModules: check modules for revisit: 0.0466 ms
visitModules: prepare: 0.6327 ms
visitModules: visiting: 7.5088 ms
2377412 queue items processed (1183284 blocks)
212 chunk groups connected
184 chunk groups processed for merging (212 module sets, 3 forked, 17887 + 0 modules forked, 25618 + 0 modules merged into fork, 17801 resulting modules)
184 chunk group info updated (0 already connected chunk groups reconnected)
visitModules: 4646.700283 ms
connectChunkGroups: 1.2897 ms
cleanup: 0.5593 ms

LOG from webpack.FileSystemInfo
74917 new snapshots created
0% root snapshot uncached (0 / 113980)
0% children snapshot uncached (0 / 0)
0 entries tested
File info in cache: 68593 timestamps 0 hashes 0 timestamp hash combinations
File timestamp snapshot optimization: 82% (538990/660274) entries shared via 2865 shared snapshots (152815 times referenced)
Directory info in cache: 1 timestamps 0 hashes 0 timestamp hash combinations
Directory timestamp snapshot optimization: 0% (0/1) entries shared via 0 shared snapshots (0 times referenced)
Missing items snapshot optimization: 73% (238747/329206) entries shared via 4374 shared snapshots (60172 times referenced)
Managed items info in cache: 0 items

LOG from webpack.Watching
done hook: 190.095494 ms
storeBuildDependencies: 3.4207 ms
beginIdle: 3.9957 ms

And if I disable any dynamic imports by using babel-plugin-dynamic-import-node then I'll get slightly less time spent in visitModules (less than 0.5 s)

Webpack logs with babel-dynamic-import

100% (31840 affected + 0 infected of 31840) modules flagged as affected (31840 new modules, 0 changed, 0 references changed, 0 unchanged, 0 were not built) compute affected modules: 296.223434 ms finish modules: 15.761897 ms report dependency errors and warnings: 307.241732 ms optimize dependencies: 0.3177 ms create chunks: 544.269179 ms 0% modules flagged as affected by chunk graph (31840 new modules, 0 changed, 0 unchanged) compute affected modules with chunk graph: 213.570052 ms optimize: 643.424658 ms 27958 modules hashed, 0 from cache (0.88 variants per module in average) module hashing: 1230.355427 ms 100% code generated (27956 generated, 0 from cache) code generation: 4562.625918 ms runtime requirements.modules: 216.952694 ms runtime requirements.chunks: 38.561599 ms runtime requirements.entries: 4.5616 ms runtime requirements: 262.439893 ms hashing: initialize hash: 0.0094 ms hashing: sort chunks: 0.1772 ms hashing: hash runtime modules: 42.491898 ms hashing: hash chunks: 422.087988 ms hashing: hash digest: 0.9766 ms hashing: process full hash modules: 0.0032 ms hashing: 470.203986 ms 100% code generated (16 generated, 0 from cache) record hash: 0.1962 ms module assets: 18.619199 ms create chunk assets: 2269.294433 ms process assets: 877.657474 ms

LOG from webpack.Compiler
make hook: 538650.482171 ms
finish make hook: 1342.499603 ms
finish compilation: 620.990163 ms
seal compilation: 11285.573424 ms
afterCompile hook: 1.5306 ms
emitAssets: 6284.810903 ms
emitRecords: 1.3997 ms

LOG from webpack.ResolverCachePlugin
29% really resolved (46771 real resolves with 0 cached but invalid, 113632 cached valid, 0 concurrent)

LOG from webpack.buildChunkGraph
visitModules: prepare: 227.601256 ms
visitModules: visiting: 474.178495 ms
56451 queue items processed (27958 blocks)
0 chunk groups connected
0 chunk groups processed for merging (0 module sets, 0 forked, 0 + 0 modules forked, 0 + 0 modules merged into fork, 0 resulting modules)
0 chunk group info updated (0 already connected chunk groups reconnected)
visitModules: 475.582295 ms
connectChunkGroups: 0.1497 ms
cleanup: 0.0644 ms

LOG from webpack.FileSystemInfo
74748 new snapshots created
0% root snapshot uncached (0 / 113632)
0% children snapshot uncached (0 / 0)
0 entries tested
File info in cache: 68593 timestamps 0 hashes 0 timestamp hash combinations
File timestamp snapshot optimization: 81% (535485/658517) entries shared via 2879 shared snapshots (148310 times referenced)
Directory info in cache: 1 timestamps 0 hashes 0 timestamp hash combinations
Directory timestamp snapshot optimization: 0% (0/1) entries shared via 0 shared snapshots (0 times referenced)
Missing items snapshot optimization: 73% (238554/327783) entries shared via 4342 shared snapshots (62039 times referenced)
Managed items info in cache: 0 items

LOG from webpack.Watching
done hook: 6.8447 ms
storeBuildDependencies: 4.1044 ms
beginIdle: 2.016 ms

visitModules is spent mostly in this code
that have the comment about possibility of removing it. @sokra do you have any plans on improvements for this code? We really rely on dynamic imports nowadays and cannot use lazy compilation due to incopatibility with SSR

sibelius · 2022-12-05T11:23:34Z

enable lazyCompilation it will make everything faster

mozhuanzuojing · 2023-03-26T11:56:43Z

enable lazyCompilation it will make everything faster

very fast

mozhuanzuojing · 2023-03-26T23:43:10Z

In the last weeks we spend a bit time optimizing rebuild performance further.

We added a new experimental flag experiments.cacheUnaffected which improves rebuild performance by detecting modules unaffected by the current change and skipping computation for them.

We added a new faster hashing algorithm output.hashFunction: "xxhash64" which is implemented in assemblyscript and a bit faster compared to the default one

We improved module.unsafeCache to be much faster

We added experiments.futureDefaults which enables experiments.cacheUnaffected, output.hashFunction: "xxhash64" and a few other things.

With experiments.futureDefaults and module.unsafeCache we now see all of our benchmarks (https://webpack.github.io/benchmark/) being faster in webpack 5 compared to webpack 4.

If that's not enough, webpack 5 has optimization.sideEffects enabled by default, while webpack 4 had that disabled by default. You can use optimization.sideEffects: false to disable that for webpack 5 too.

Need more even more speed? You can also use optmization.providedExports: false to disable another expensive step in the build. (This might break some apps which use complex circular ESM reexports) (We eventually add caching for that step to cacheUnaffected, which would make it fast without the need to disable it)

Great work

sibelius · 2023-10-12T22:04:59Z

can we close this ?

sibelius · 2023-10-12T22:05:25Z

check this tips https://dev.to/woovi/web-dev-efficiency-achieving-faster-builds-and-lower-memory-load-with-webpack-4cm8

alexander-akait · 2023-12-19T15:41:55Z

I think yes, we can close, if somebody encounters to perfomance problems - feel free to open a new issue

JerryLyu93 · 2024-02-04T08:23:43Z

We noticed the big build/rebuild slowdowns on the big applications with a lot of dynamic imports. The most noticeable time is spent in visitModules block (more than 4s).

Webpack logs with dynamic imports
And if I disable any dynamic imports by using babel-plugin-dynamic-import-node then I'll get slightly less time spent in visitModules (less than 0.5 s)

Webpack logs with babel-dynamic-import
visitModules is spent mostly in this code that have the comment about possibility of removing it. @sokra do you have any plans on improvements for this code? We really rely on dynamic imports nowadays and cannot use lazy compilation due to incopatibility with SSR

Did you resolved this not by use babel-plugin-dynamic-import-node?

alexander-akait closed this as completed Dec 3, 2020

sokra reopened this Dec 3, 2020

webpack-bot added webpack-5 performance labels Dec 3, 2020

sokra mentioned this issue Dec 4, 2020

improve rebuild performance #12121

Merged

eulier1 mentioned this issue Jul 1, 2022

Qiankun-main-app collabdash/nuxt3-micro-frontend#2

Open

alexander-akait closed this as completed Dec 19, 2023

[webpack 5] rebuild is twice as slow comparing to webpack 4 #12102

[webpack 5] rebuild is twice as slow comparing to webpack 4 #12102

Comments

meskill commented Dec 3, 2020 • edited Loading

alexander-akait commented Dec 3, 2020

Thebarda commented Dec 3, 2020

sokra commented Dec 3, 2020

alexander-akait commented Dec 3, 2020

Thebarda commented Dec 3, 2020

alexander-akait commented Dec 3, 2020 • edited Loading

alexander-akait commented Dec 3, 2020

alexander-akait commented Dec 3, 2020

sokra commented Dec 4, 2020

sokra commented Dec 4, 2020 • edited Loading

meskill commented Dec 4, 2020

keepitterron commented Dec 6, 2020

alexander-akait commented Dec 7, 2020

keepitterron commented Dec 7, 2020

keepitterron commented Dec 7, 2020

joshxyzhimself commented Dec 11, 2020

alexander-akait commented Dec 11, 2020

jolyndenning commented Dec 11, 2020

joshxyzhimself commented Dec 12, 2020

joshxyzhimself commented Dec 12, 2020

sokra commented Dec 12, 2020

meskill commented Dec 16, 2020 • edited Loading

alexander-akait commented Dec 16, 2020

alexander-akait commented Jan 18, 2021 • edited Loading

alexander-akait commented Jan 18, 2021

alexander-akait commented Dec 30, 2021

xyxiao001 commented Dec 30, 2021

xyxiao001 commented Dec 30, 2021

alexander-akait commented Dec 30, 2021

alexander-akait commented Dec 30, 2021 • edited Loading

xyxiao001 commented Dec 30, 2021 • edited Loading

xyxiao001 commented Dec 30, 2021 • edited Loading

xyxiao001 commented Dec 30, 2021

alexander-akait commented Dec 30, 2021

VishalBhandare-Apertumo commented Apr 10, 2022

alexander-akait commented Apr 11, 2022

VishalBhandare-Apertumo commented Apr 11, 2022

alexander-akait commented Apr 11, 2022

FishOrBear commented Apr 21, 2022

vankop commented Apr 21, 2022

noppa commented Apr 21, 2022 • edited Loading

FishOrBear commented Apr 22, 2022

FishOrBear commented Apr 22, 2022 • edited Loading

sibelius commented Jul 12, 2022

alexander-akait commented Jul 12, 2022

meskill commented Dec 5, 2022

sibelius commented Dec 5, 2022

mozhuanzuojing commented Mar 26, 2023

mozhuanzuojing commented Mar 26, 2023

sibelius commented Oct 12, 2023

sibelius commented Oct 12, 2023

alexander-akait commented Dec 19, 2023

JerryLyu93 commented Feb 4, 2024

meskill commented Dec 3, 2020 •

edited

Loading

alexander-akait commented Dec 3, 2020 •

edited

Loading

sokra commented Dec 4, 2020 •

edited

Loading

meskill commented Dec 16, 2020 •

edited

Loading

alexander-akait commented Jan 18, 2021 •

edited

Loading

alexander-akait commented Dec 30, 2021 •

edited

Loading

xyxiao001 commented Dec 30, 2021 •

edited

Loading

xyxiao001 commented Dec 30, 2021 •

edited

Loading

noppa commented Apr 21, 2022 •

edited

Loading

FishOrBear commented Apr 22, 2022 •

edited

Loading