Speeding up hot-reloads of webpack-dev-server

[MarkyMarkMcDonald]

Versions:
Webpack version: 5.91.0
(dev-server and dev-middleware are also at latest released versions).
Node version: 20.5.1

Problem
Our webpack configuration is for 259 entrypoints. We use webpack-dev-server in development and making any change takes 6-8 seconds to recompile, even when only adding or removing a blank line to a single file. I took a node profile and things that stick out are buildChunkGraph taking 1.7 seconds and SplitChunksPlugin taking 2.5 seconds.

Profile: webpack-51-hot-empty-lines.cpuprofile

If I change our config to only use 20 entrypoints then things are much faster. It seems like the reload is redoing a bunch of work considering no imports are being changed (so there shouldn't be any changes to chunking) - is there any way I can configure webpack to say "hey don't try to change / recalculate chunks as part of a hot reload"?

I'd also welcome any advice on what to look into further or ways to get more actionable logs / profiles.

[alexander-akait]

Hello, do you use cache? buildChunkGraph should be faster if you enable it

Maybe we can cache some things in SplitChunksPlugin to make it more faster too

Also I see you use FlagDependecyExportsPlugin, it is require time to optimize some things, do you really need this for development purposes?

[MarkyMarkMcDonald]

hi, thanks for taking a look @alexander-akait!

We do use cache - I turned it off to confirm and the reloads were taking ~50 seconds.

Looks like FlagDependecyExportsPlugin is on by default. I turned it off and it saves minimal time (50ms, there's only about 10 files we have that use export * from) - I think it’s just showing up at the top of the stack trace because it calls compilation.hooks.finishModules.tapAsync.

I could see caching things in SplitChunksPlugin speeding things up, maybe starting from the chunk groups of the previous compile and then recalculating groups with changed modules would be low hanging fruit.

Here's the result of calling require('util').inspect with our webpack.config.js (I trimmed most of the entrypoints):

{
  devtool: 'source-map',
  mode: 'development',
  target: 'web',
  entry: {
    zendesk_web_widget: '/Users/mark.mcdonald/workspace/greenhouse/app/webpack/javascripts/zendesk_web_widget.ts',
    workday_sync_service_calls: '/Users/mark.mcdonald/workspace/greenhouse/app/webpack/javascripts/workday_sync_service_calls.ts',
    workday_sync_reports: '/Users/mark.mcdonald/workspace/greenhouse/app/webpack/javascripts/workday_sync_reports.ts',
    // <TRIMMED>
    account_contacts: '/Users/mark.mcdonald/workspace/greenhouse/app/webpack/javascripts/account_contacts.js',
    explicit: [
      '/Users/mark.mcdonald/workspace/greenhouse/node_modules/@grnhse/wrailspack/javascript/browser/data_injector.js',
      'core-js/es/object',
      'shared/utils/rollbar_logger.js',
      '@grnhse/wrailspack/javascript/browser/webpack_progress'
    ]
  },
  output: {
    filename: '[name].js',
    path: '/Users/mark.mcdonald/workspace/greenhouse/public/assets/webpack',
    hashFunction: undefined,
    publicPath: '/assets/webpack/'
  },
  resolve: {
    extensions: [ '.ts', '.tsx', '.js', '.jsx', '.js.jsx' ],
    modules: [
      'node_modules',
      '/Users/mark.mcdonald/workspace/greenhouse/app/webpack/javascripts'
    ],
    symlinks: false,
    fallback: {
      stream: '/Users/mark.mcdonald/workspace/greenhouse/node_modules/stream-browserify/index.js',
      buffer: 'buffer',
      vm: '/Users/mark.mcdonald/workspace/greenhouse/node_modules/vm-browserify/index.js',
      process: '/Users/mark.mcdonald/workspace/greenhouse/node_modules/process/browser.js',
      'process/browser': '/Users/mark.mcdonald/workspace/greenhouse/node_modules/process/browser.js'
    },
    alias: {
      svg: '/Users/mark.mcdonald/workspace/greenhouse/app/assets/images/svg',
      images: '/Users/mark.mcdonald/workspace/greenhouse/app/assets/images',
      jquery: '/Users/mark.mcdonald/workspace/greenhouse/app/webpack/javascripts/legacy/jquery'
    },
    cache: true
  },
  optimization: {
    runtimeChunk: 'single',
    minimizer: [],
    splitChunks: {
      maxInitialRequests: 10,
      name: false,
      chunks: 'all',
      minChunks: 2,
      cacheGroups: {
        jsAssets: {
          test: /[\\/]app[\\/]assets[\\/]javascripts[\\/](3rdparty|shared)[\\/]/,
          priority: 10,
          name: 'js-assets'
        },
        legacy: {
          test: /[\\/]app[\\/]webpack[\\/]javascripts[\\/]legacy[\\/]/,
          priority: 10,
          name: 'js-legacy'
        },
        tinymce: {
          test: /[\\/]node_modules[\\/]tinymce|tinymce-word-paste-filter[\\/]|[\\/]app[\\/]webpack[\\/]javascripts[\\/]tinymce[\\/]/,
          priority: 20,
          name: 'vendor-tinymce'
        },
        vendor: {
          priority: 5,
          test: /[\\/]node_modules[\\/]/,
          minSize: 20000,
          minSizeReduction: 50000,
          reuseExistingChunk: true
        },
        head: {
          priority: 15,
          name: 'head-common',
          reuseExistingChunk: true,
          enforce: true,
          chunks: [Function: chunks],
          minChunks: 1
        },
        default: { chunks: 'initial' }
      }
    },
    providedExports: false
  },
  performance: { hints: false },
  plugins: [
    ProvidePlugin { definitions: { React: 'react' } },
    ProvidePlugin { definitions: { Buffer: [ 'buffer', 'Buffer' ] } },
    ProvidePlugin { definitions: { process: 'process/browser' } },
    StatsWriterPlugin {
      opts: {
        filename: 'manifest.json',
        fields: [ 'entrypoints', 'errors' ],
        stats: { all: false, entrypoints: true, errors: true },
        transform: [Function: DEFAULT_TRANSFORM],
        emit: true
      }
    },
    DefinePlugin {
      definitions: {
        __DEV__: 'true',
        __TEST__: false,
        __PROD__: 'false',
        'process.env': { NODE_ENV: '"development"' },
        __SETUP_TAG_ID__: '"__webpack_config__"',
        __GIT_SHA__: '""'
      }
    },
    MiniCssExtractPlugin {
      _sortedModulesCache: WeakMap { <items unknown> },
      options: {
        filename: '[name].css',
        ignoreOrder: false,
        experimentalUseImportModule: undefined,
        runtime: true,
        chunkFilename: '[name].css'
      },
      runtimeOptions: {
        insert: undefined,
        linkType: 'text/css',
        attributes: undefined
      }
    },
    IgnorePlugin {
      options: { resourceRegExp: /^\.\/locale$/, contextRegExp: /moment$/ },
      checkIgnore: [Function: bound checkIgnore]
    },
    ReactRefreshPlugin {
      options: {
        overlay: false,
        exclude: /node_modules/i,
        include: /\.([cm]js|[jt]sx?|flow)$/i
      }
    },
    DefinePlugin { definitions: { __ENABLE_REDUX_LOGGER__: 'true' } }
  ],
  module: {
    rules: [
      {
        loader: 'esbuild-loader',
        test: /\.(es6|js|jsx)$/,
        options: { target: 'es2015', loader: 'jsx' },
        exclude: [
          /node_modules\/(?!(@grnhse\/wrailspack\/javascript\/browser|@grnhse\/datacoral_tracker)\/)/i
        ]
      },
      {
        loader: 'esbuild-loader',
        test: /\.(ts|tsx)$/,
        options: { target: 'es2015', loader: 'default' },
        exclude: [
          /node_modules\/(?!(@grnhse\/wrailspack\/javascript\/browser|@grnhse\/datacoral_tracker)\/)/i
        ]
      },
      {
        test: /\.mod\.css$/,
        use: [
          '/Users/mark.mcdonald/workspace/greenhouse/node_modules/@grnhse/wrailspack/node_modules/mini-css-extract-plugin/dist/loader.js',
          { loader: 'css-loader', options: { modules: true } }
        ]
      },
      {
        test: /\.css$/,
        exclude: /\.mod\.css$/,
        use: [
          '/Users/mark.mcdonald/workspace/greenhouse/node_modules/@grnhse/wrailspack/node_modules/mini-css-extract-plugin/dist/loader.js',
          { loader: 'css-loader', options: { modules: false } }
        ]
      },
      {
        test: /.(yml$|\w+.yml$|yaml$)/,
        use: [ 'json-loader', 'yaml-loader' ]
      },
      { test: /\.(png|jpeg|jpg|gif|svg|ttf)$/, type: 'asset/resource' },
      { test: /\.html$/i, loader: 'html-loader' },
      { test: /\.m?js/, resolve: { fullySpecified: false } }
    ]
  },
  devServer: {
    static: false,
    compress: true,
    hot: true,
    liveReload: true,
    client: {
      overlay: {
        errors: true,
        warnings: true,
        runtimeErrors: [Function: runtimeErrors]
      },
      progress: true,
      reconnect: false,
      webSocketURL: 'ws://localhost:3808/ws',
      webSocketTransport: 'ws'
    },
    port: 3808,
    allowedHosts: 'all',
    headers: { 'Access-Control-Allow-Origin': '*' },
    webSocketServer: 'ws'
  },
  resolveLoader: {
    modules: [ 'node_modules/@grnhse/wrailspack/node_modules', 'node_modules' ]
  },
  ignoreWarnings: [ /Conflicting order. Following module has been added/ ],
  cache: true
}

[alexander-akait]

I could see caching things in SplitChunksPlugin speeding things up, maybe starting from the chunk groups of the previous compile and then recalculating groups with changed modules would be low hanging fruit.

Yeah, do you want to send a PR? I am glad to see it

Also we have https://webpack.js.org/configuration/experiments/#experimentslazycompilation, do you try this?

[MarkyMarkMcDonald]

Yeah, do you want to send a PR? I am glad to see it

👍 I'll dig in and see what I can come up with

experimentslazycompilation

I tried enabling lazy compilation. It booted successfully (and almost instantly) but the first two times I would try to load a page it failed - I found an error in the console about receiving html when expecting json but couldn't get a non-minified trace and will need to dig in further. I'll try to track down the root cause an open an issue - I'm guessing there's some edge case with handling the first time a chunk is requested - our setup isn't the most typical. We don't use the HTML plugin as we have a rails monolith - we have some ruby code that gets the dev-server's manifest and determines what script tags should be put on a page based on what entrypoints are asked for by our views. So it might be related to that.

I've played with running two dev-servers, one with everything and one with a limited set of entrypoints. Then having our server know if it can use the faster dev-server depending on the page you're visiting. It works pretty well, but I'd rather get lazy compilation working than add this complexity to our development workflow.

[alexander-akait]

@MarkyMarkMcDonald Maybe you can create a minimum reprducible example so I will look and try to help you?