Transition to webpack 5 #4982

TheTrio · 2022-05-23T05:23:12Z

What this PR does

Upgrades webpack to version 5

Changes

1. Switch to relative paths for css assets

This is because absolute paths are now relative to the root of the project.

2. Using asset modules instead of `url-loader`

url-loader has been deprecated for Webpack 5 in favor of asset modules. There are multiple asset types and I have gone with the general asset type. From the documentation -

Now webpack will automatically choose between resource and inline by following a default condition: a file with size less than 8kb will be treated as a inline module type and resource module type otherwise.

3. Single manifest file

Before, two manifest files were being created - one for css and another for JS. I've combined them into one manifests.json file so that the manifest plugin is only executed once.

4. Replace `webpack-rtl-plugin` with `rtlcss-webpack-plugin`

webpack-rtl-plugin was last updated on Mar 15, 2019. Despite there being a PR which adds support for Webpack 5, it hasn't been merged for over 8 months. So its been replaced with rtlcss-webpack-plugin which looks to be maintained more actively.

5. Add CssMinimizerWebpackPlugin to minify css

For some reason,webpack-rtl-plugin also minified the css files - both the RTL and non RTL ones. Since we no longer use that plugin, we need to minify the css with css-minimizer-webpack-plugin

6. Remove ExcludeAssetsPlugin

This plugin was used to clean up the empty JS files which were being generated by Webpack. With v5, Webpack no longer generates these files, so there was no need to use this plugin.

7. Remove Parallel Webpack

Parallel Webpack was used to speed up the build process by spreading the workload across multiple processors. The repo for this project has been archived and it does not support webpack 5. Webpack doesn't have native support for parallelism, but that might change in the future.

8. Use webpack dev server by default

Earlier, this was only being used with yarn hot. I couldn't get yarn hot to work so it might have been broken for a while. Regardless, yarn start now uses webpack dev server. The files are written to the disk presently, but I will try to use the server directly. I haven't been been able to make that work so far. The switch to webpack dev server also means that the webpage reloads automatically on code changes. It also shows warnings and errors inside of the browser.

9. Fix regex for coverage

The regex which split main.js into multiple modules broke because webpack uses arrow functions instead of the function keyword. I've changed that so that yarn coverage still works as expected.

There are some other configuration changes but these are the major ones I could think of.

Performance

These numbers are obviously not scientific but the time taken is consistent from the testing I've done.

command	cache	Webpack 5	Webpack 4
`yarn start`	no	70s	40s
`yarn start`	yes	10s	10s
`yarn build`	no	101s	75s
`yarn build`	yes	41s	25s

For rebuilds, it takes about 2000ms the first time and the subsequent rebuilds are much faster, around 800ms. This is probably due to the source map configuration. Unfortunately, if we want the right line numbers in the error messages, I think this is the best that can be done. More details can be found here

It looks like webpack now generates arrow functions instead of using the function keyword. This, along with a change in the source map configuration, broke yarn coverage. I have updated the regex and switched back to the old source map configuration for the time being. This should hopefully suffice

This package hasn't been updated for webpack 5

The webpack-rtl-plugin hasn't been updated for 3 years and doesn't support webpack 5. This switch also means that we have to minify the css ourselves(which was earlier done by the rtl plugin) using css-minimizer-webpack-plugin

See https://webpack.js.org/loaders/babel-loader/#options

ragesoss · 2022-05-24T19:01:43Z

Super helpful summary! Can you provide before/after numbers (ie, v4 vs v5) for the yarn commands?

TheTrio · 2022-05-24T19:45:42Z

Edited. I think the main reason for the performance difference is because of parallel webpack running things in, well, parallel.

TheTrio · 2022-05-24T22:11:20Z

Here are the new numbers

command	cache	Webpack 5	Webpack 4
`yarn start`	no	20s	40s
`yarn start`	yes	10s	10s
`yarn build`	no	58s	75s
`yarn build`	yes	41s	25s

It looks like ESLint is the culprit here. I've enabled lintDirtyModulesOnly which caches dirty modules only. So while the initial build times are faster, during development, when you edit the file for the first time, it will take time - 3-7 seconds. Subsequent edits will be faster and as the cache builds up, it will drop to around 1000ms.

Essentially what we were doing during the build process(building up the eslint cache) is now being done on demand - when you edit files. This slows initial edits during development.

Spawning threads during development slowed down the linting. So its now enabled only during production builds. There's also a new command for disabling linting altogether

TheTrio · 2022-05-25T17:26:06Z

Since most editors and IDEs run ESlint in the background anyways, I've added a new command - start:no-lint. This disables the ESlint plugin - which means that even the first edit is compiled instantly.

yarn start is slow on the first edit, but after that(as the cache builds up), it should be almost as fast as yarn start:no-lint

dhruvdutt · 2022-06-06T14:38:36Z

css.webpack.js

        }
      ]
    },
    plugins: [
      // extracts CSS to a separate file
      new MiniCssExtractPlugin({
-        filename: env.development ? '../stylesheets/[name].css' : '../stylesheets/[name].[hash].css',
+        filename: env.development ? '../stylesheets/[name].css' : '../stylesheets/[name].[contenthash].css',


What other hash options are available?

Looks like the supported substitutions are hash, fullhash, contenthash and chunkhash. The reason I went with contenthash was because the webpack 5 migration guide recommended it

dhruvdutt · 2022-06-06T14:39:42Z

css.webpack.js

-      // node environment
-      new webpack.DefinePlugin({
-        'process.env': {
-          NODE_ENV: JSON.stringify(mode),


Is this available in webpack 5?

It is but it looks like since Webpack 4 defining mode should set process.env.NODE_ENV automatically - so this might've been redundant as well.

dhruvdutt · 2022-06-06T14:40:04Z

css.webpack.js

-        filename: '../stylesheets/rtl-[name].[contenthash].css'
+      // this is done only in production/coverage
+      ...(env.production || env.coverage ? [new RtlCssPlugin({
+        filename: '../stylesheets/rtl-[name].[fullhash].css'


What other hash options are available?

The rtl-css-plugin still has some deprecation warnings when using contenthash. Which is why I used fullhash here.

dhruvdutt · 2022-06-06T14:42:35Z

webpack.config.js

+    output.devServer = {
+      devMiddleware: {
+        publicPath: path.join(__dirname, '/public'),
+        writeToDisk: true,


Why don't we keep this in memory itself? Why do we need to write to disk?

currently has an issue with dynamic import bundles if we switch to in-memory

dhruvdutt · 2022-06-06T14:43:54Z

js.webpack.js

    optimization: {
      splitChunks: {
        cacheGroups: {
-          vendors: {
+          defaultVendors: {


Can we explore doing splitting for vendor bundle?

dhruvdutt · 2022-06-06T15:23:53Z

css.webpack.js

+    optimization: {
+      minimizer: [
+        '...',
+        new CssMinimizerPlugin(),


Explore if we can pass any options that might be helpful.

@TheTrio Were you able to explore this?

The defaults were good enough in my opinion so there wasn't much to configure. As for CSS Nano options, auto prefixing was the only one that felt necessary to enable but then I found out that stylus loader already does that.

So no, not much there I think. Do you have some specific configuration in mind?

dhruvdutt · 2022-06-06T15:33:07Z

js.webpack.js

@@ -23,39 +22,41 @@ const entry = {
 };

 module.exports = (env) => {
-  const doHot = env.development && !env.watch_js;


Need to check how this works with Ruby server and enable hot module replacement.

dhruvdutt · 2022-06-06T15:37:05Z

js.webpack.js

 const LodashModuleReplacementPlugin = require('lodash-webpack-plugin');
 const ESLintPlugin = require('eslint-webpack-plugin');
 const MomentLocalesPlugin = require('moment-locales-webpack-plugin');
-const webpack = require('webpack');

 const jsSource = `./${config.sourcePath}/${config.jsDirectory}`;
 const appRoot = path.resolve('./');


Remove this.

dhruvdutt · 2022-06-06T15:37:12Z

js.webpack.js

-    ? path.resolve(appRoot, `${config.outputPath}/${config.jsDirectory}`)
-    : path.resolve(`${config.outputPath}/${config.jsDirectory}`);
-  const devtool = env.coverage ? 'cheap-module-source-map' : 'cheap-eval-source-map';
+  const outputPath = path.resolve(appRoot, `${config.outputPath}/${config.jsDirectory}`);


Suggested change

const outputPath = path.resolve(appRoot, `${config.outputPath}/${config.jsDirectory}`);

const outputPath = path.resolve(`${config.outputPath}/${config.jsDirectory}`);

dhruvdutt · 2022-06-06T15:41:50Z

js.webpack.js

-  const devtool = env.coverage ? 'cheap-module-source-map' : 'cheap-eval-source-map';
+  const outputPath = path.resolve(appRoot, `${config.outputPath}/${config.jsDirectory}`);
+
+  let devtool;


Suggested change

let devtool;

let devtool = 'eval-cheap-source-map';

dhruvdutt · 2022-06-06T15:53:09Z

js.webpack.js

    optimization: {
      splitChunks: {
        cacheGroups: {
-          vendors: {
+          defaultVendors: {


Can explore later on how to split vendor checks and make it work with Ruby

dhruvdutt · 2022-06-06T15:59:02Z

lib/tasks/coverage.rake

@@ -14,7 +14,7 @@ namespace :generate do
      funs.each do |fun|
        counter += 1
        text = text.sub(fun, "*/module_#{counter}") # Replaces the function with module_{number}
-        fun = fun.gsub(%r/\/[*]{3}\/ \(function/, "var module_#{counter} = (function")
+        fun = fun.gsub(%r/\/[*]{3}\/ \(/, "var module_#{counter} = (")


@ragesoss Need to validate if coverage.rake file is working as expected since the output in public/assets/javascripts/main.js might have changed from webpack 4 to 5.

Yep, I tried it out locally and the coverage.rake is still working.

ragesoss · 2022-06-07T17:44:57Z

@TheTrio @dhruvdutt let me know when both of you are happy with this PR. I can merge it as soon as you both think it's ready.

dhruvdutt · 2022-06-07T18:08:55Z

Let's try to get rid of js.webpack and css.webpack and keep only webpack.js and probably a different file for webpack plugins if it's too large to put in the same file.
Keeping js and css files differently is an anti-pattern and the biggest problem is conflicts while merging both these objects. The earlier version had overrides, conflicting rules, objects and variables, all of which make it difficult to visualize the final config that's going to get applied.
IMO having one large file organized and documented inline is better than different files which merge with conflicts and ambiguous override states.

TheTrio · 2022-06-07T19:20:56Z

Yeah having just one webpack config would definitely make things simpler. I'll get that done by tomorrow.

Apart from that, I don't think there's anything major left to do in this PR. There's still some stuff I want to explore - like HMR, speeding up eslint and switching to memory mode for the dev server.

But I think those can be tracked independently of this PR.

dhruvdutt · 2022-06-09T15:09:22Z

@TheTrio Can you please create issues for things we're going to be looking into separately?

HMR
eslint optimizations
Webpack dev server in-memory chunks
Vendor chunking

transition to webpack 5

55702db

TheTrio force-pushed the develop branch from 5c5dc94 to 55702db Compare May 23, 2022 06:35

TheTrio added 6 commits May 23, 2022 14:47

fix source maps

3c9eab5

Remove parallel webpack

19d1354

This package hasn't been updated for webpack 5

Switch to rtlcss-webpack-plugin

7b578ca

The webpack-rtl-plugin hasn't been updated for 3 years and doesn't support webpack 5. This switch also means that we have to minify the css ourselves(which was earlier done by the rtl plugin) using css-minimizer-webpack-plugin

Set compression of babel cache to false

cad1962

See https://webpack.js.org/loaders/babel-loader/#options

Clean up the configs

82062b3

TheTrio marked this pull request as ready for review May 24, 2022 15:44

TheTrio added 2 commits May 25, 2022 03:10

Updated lodash plugin

b1e3a4c

enable threads for eslint

4610c83

Disable threads during development and add option to disable linting

fc50cfb

Spawning threads during development slowed down the linting. So its now enabled only during production builds. There's also a new command for disabling linting altogether

TheTrio added 2 commits June 4, 2022 01:30

Merge branch 'master' into develop

98fe934

resolve conflicts

482a1c4

dhruvdutt reviewed Jun 6, 2022

View reviewed changes

TheTrio added 2 commits June 8, 2022 08:43

Merged CSS and JS webpack configs

f546e86

Show progress during build and coverage

757ca33

dhruvdutt approved these changes Jun 9, 2022

View reviewed changes

ragesoss merged commit 485ef7a into WikiEducationFoundation:master Jun 9, 2022

TheTrio deleted the develop branch June 9, 2022 15:34

TheTrio mentioned this pull request Jun 10, 2022

Optimize ESLint #5004

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Transition to webpack 5 #4982

Transition to webpack 5 #4982

TheTrio commented May 23, 2022 •

edited

ragesoss commented May 24, 2022

TheTrio commented May 24, 2022

TheTrio commented May 24, 2022 •

edited

TheTrio commented May 25, 2022

dhruvdutt Jun 6, 2022

TheTrio Jun 7, 2022 •

edited

dhruvdutt Jun 6, 2022

TheTrio Jun 7, 2022

dhruvdutt Jun 6, 2022

TheTrio Jun 7, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 9, 2022

TheTrio Jun 9, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

dhruvdutt Jun 6, 2022

ragesoss Jun 6, 2022

ragesoss commented Jun 7, 2022

dhruvdutt commented Jun 7, 2022

TheTrio commented Jun 7, 2022 •

edited

dhruvdutt commented Jun 9, 2022

	const outputPath = path.resolve(appRoot, `${config.outputPath}/${config.jsDirectory}`);
	const outputPath = path.resolve(`${config.outputPath}/${config.jsDirectory}`);

Transition to webpack 5 #4982

Transition to webpack 5 #4982

Conversation

TheTrio commented May 23, 2022 • edited

What this PR does

Changes

1. Switch to relative paths for css assets

2. Using asset modules instead of url-loader

3. Single manifest file

4. Replace webpack-rtl-plugin with rtlcss-webpack-plugin

5. Add CssMinimizerWebpackPlugin to minify css

6. Remove ExcludeAssetsPlugin

7. Remove Parallel Webpack

8. Use webpack dev server by default

9. Fix regex for coverage

Performance

ragesoss commented May 24, 2022

TheTrio commented May 24, 2022

TheTrio commented May 24, 2022 • edited

TheTrio commented May 25, 2022

Choose a reason for hiding this comment

TheTrio Jun 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ragesoss commented Jun 7, 2022

dhruvdutt commented Jun 7, 2022

TheTrio commented Jun 7, 2022 • edited

dhruvdutt commented Jun 9, 2022

TheTrio commented May 23, 2022 •

edited

2. Using asset modules instead of `url-loader`

4. Replace `webpack-rtl-plugin` with `rtlcss-webpack-plugin`

TheTrio commented May 24, 2022 •

edited

TheTrio Jun 7, 2022 •

edited

TheTrio commented Jun 7, 2022 •

edited