add experimental lazy compilation

[sokra]

This adds an experimental feature which allows to compile only the part of the application which is currently in use. Additional import()s and entrypoints which are currently not in use do not affect compilation performance when this feature is enabled.

When used webpack opens an additional communication channel between client and compiler via Server-Send-Events. With this channel the currently used import()s are told the compiler. The compiler will include these import()s resp. entrypoints in the next compilation. When in watch mode and new import()s/entrypoints are communicated to the compiler, it will automatically recompile.

When Hot Module Replacement is not enabled, (manual) refresh of the page is required.

This also works when targeting node.js. Including HMR in node.js.

After 2 minutes not using a import() resp. entrypoint it be garbage collected. Once compiled modules will stay in cache, so compiling it again is probably pretty fast. It's recommended to combine this with the Persistent Cache to keep the cache even after restarting.

For logging use infrastructureLogging.debug: /LazyCompilationBackend/.

What kind of change does this PR introduce?
feature

Did you add tests for your changes?
yes, but that could be improved.

Does this PR introduce a breaking change?
no

What needs to be documented once your changes are merged?

• add experiments.lazyCompilation: Compiles import() only when requested using HMR.
• add experiments.lazyCompilation.client: Allows to use a custom client. It need to expose a single export named keepAlive which is a ({ data: any, active: boolean, module: Module, onError: (err: Error) => void }) => () => void. It's returning a dispose function. As default webpack/hot/lazy-compilation-{node|web}.js is used.
  • When writing your own client best import the default client with require("webpack/hot/lazy-compilation-web.js" + __resourceQuery) and call the keepAlive method from your own implementation.
  • Custom client might be useful if you want to add some UI effects while waiting for compilation.
• add experiments.lazyCompilation.backend: Allows to use a custom backend.
  • (compiler: Compiler, client: string) => Promise<Backend> (Promise or node-style callback)
  • Backend = { dispose: (callback) => void, module: (module: WebpackModule) => { client: string, data: any, active: boolean } }
  • The default backend passes a URL prefix to a EventSource server as query string to the client. data is a url encoded key for the module. urlPrefix + keys.join("@") should be used to access the EventSource.
• see above for more details

---

Prior art

• next.js detects the used routes and adds entrypoint on demand, which makes unused routes not influencing the compilation
• https://github.com/liximomo/lazy-compile-webpack-plugin
  • uses timestamp write to disk to invalidate build
  • uses a loader instead of a separate Module + Chunk

cc @liximomo

[webpack-bot]

For maintainers only:

• This needs to be documented (issue in webpack/webpack.js.org will be filed when merged)
• This needs to be backported to webpack 4 (issue will be created when merged)

[alexander-akait]

What about https?

[sokra]

That would require providing a cert and key. We can definitely add that before this leaves experimental status.

[liximomo]

I am so happy that webpack will integrate this feature into its core. 🎉 🎉 🎉

[webpack-bot]

Thank you for your pull request! The most important CI builds succeeded, we’ll review the pull request soon.

[webpack-bot]

I've created an issue to document this in webpack/webpack.js.org.

[sanriqing]

it's so cooooool!

[sokra]

Great that you like it. Could you provide more feedback/details?

[sanriqing]

Great that you like it. Could you provide more feedback/details?

Sure! i will test it in weekend and give some feedback.

[pcostanz]

Enabling this feature seems to cause our (very large) build to take longer, reload times on a small react component seem unchanged. The app has many (hundreds) of dynamic import chunks. Let me know if I can provide any other info.

Relevant dependencies:

    "@pmmmwh/react-refresh-webpack-plugin": "^0.4.3",
    "express": "^4.17.1",
    "react-refresh": "^0.8.3",
    "webpack": "^5.21.2",
    "webpack-bundle-analyzer": "^4.3.0",
    "webpack-dev-middleware": "^4.1.0",
    "webpack-hot-middleware": "^2.25.0",

Before

webpack 5.21.2 compiled with 1 warning in 128680 ms
webpack building...
webpack built 64dcfd39f0279a778a78 in 6257ms

After

webpack 5.21.2 compiled with 1 warning in 202855 ms
webpack building...
webpack built 9fb56e85ce36e9f5b063 in 6227ms

[alexander-akait]

webpack-hot-middleware doesn't work with this