Webpack is slow to start and always performs work, even if it is unnecessary

[kevinburke]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

If the --watch mode is not specified, Webpack always runs the function in the config file on the inputs, to produce the output.

If the current behavior is a bug, please provide the steps to reproduce.

1. Specify a Webpack function that transforms the inputs into an output, in a config file.
2. Make a change to one of the input files and save it.
3. Invoke webpack with webpack --config /path/to/config.json, to write the output file as a function of the inputs.
4. Immediately run webpack again with the same commands.

What is the expected behavior?

The second time webpack is invoked, I expect it to detect that the mtimes of the inputs are all older than the mtimes of the output file, print that there is no work to be done, and exit with a 0 return code.

If this is a feature request, what is motivation or use case for changing the behavior?

Make is a build automation tool released for UNIX systems in April 1976, 41 years ago. If I define a Make target as follows:

output.js: static/one.js static/two.js

and invoke it with:

make output.js

Make will check the mtimes of static/one.js, static/two.js, and output.js (which you can check by using the Lstat syscall on the affected files). If the mtime for output.js is newer than its inputs, make outputs the following:

make: Nothing to be done for `output.js'.

And exits with a 0 return code. This means it is faster - make does not do unnecessary work - and more composable. It is always easy to invoke make output.js as part of another build script, since make will exit early if there is no work to be done.

By contrast, webpack does all of the work every single time. This makes it more difficult to compose as part of a larger build process, since invoking webpack means the work is always performed even if there is no point in doing the work since the inputs haven't changed since the output file was written.

In addition to doing unnecessary work, webpack has a slower startup time than make. Here is the amount of time it takes make to print the version string:

$ time make -v
GNU Make 4.2.1
Built for x86_64-apple-darwin16.6.0
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
make -v  0.00s user 0.00s system 32% cpu 0.026 total

Or 26 milliseconds. It takes webpack, released 36 years later, about 8 times as long to print the version string:

$ time webpack -v
2.6.1
webpack -v  0.17s user 0.04s system 84% cpu 0.242 tota

This is a significant performance regression, both in startup time and in the amount of work that is performed. It's especially confusing given the modest improvements in CPU performance, and the increases in available RAM, in the years between 1976 and 2012.

It would be nice if performance improved along one or both of these axes.

[marcintustin]

Why not add this as a flag in a PR?

[marcintustin]

Why not add this as a flag in a PR?

[TheLarkInn]

This is although a little jabby 😂, still an excellent analysis. Given this is the case, would you considering looking more in-depth at GitHub.com/webpack/watchpack which contains the underlying watching functionality to see if are easy wins. It leverages chokidar behind the scenes.

[TheLarkInn]

This is although a little jabby 😂, still an excellent analysis. Given this is the case, would you considering looking more in-depth at GitHub.com/webpack/watchpack which contains the underlying watching functionality to see if are easy wins. It leverages chokidar behind the scenes.

[TheLarkInn]

But then again we consciously purge the inputFileSystem cache for non watching builds. Something to consider: if you are using unique hashing and constantly writing this to in memory fs cache, your cache will continue to grow. Make although a shining example of a primitive fast task runner does maybe 1/100 of the operational logic that webpack is performing. Cases we might need to be aware of that would need to invalidate cache:

• almost any change to the configuration could allow for discovery of new entries files etc, we could no use the lstat or mtimes of previous build in this case.
• hashing of any kind (unless maybe permissable from the user) would cause our mem fs cache to continue to grow
• loader caching: the operation of one instance of a loader is likely comparable to a task in make is performing, with this in mind, we have already created a loader that will help users cache expensive loader operations.

I'm sure the list goes on. Regardless we are always looking for individuals to help performance tune all of the facets of our ecosystem so if you are interested in doing a more comparable, and far less hyperbolic analysis, then we are more than willing to lend a helping hand guiding you through the architecture

[TheLarkInn]

But then again we consciously purge the inputFileSystem cache for non watching builds. Something to consider: if you are using unique hashing and constantly writing this to in memory fs cache, your cache will continue to grow. Make although a shining example of a primitive fast task runner does maybe 1/100 of the operational logic that webpack is performing. Cases we might need to be aware of that would need to invalidate cache:

• almost any change to the configuration could allow for discovery of new entries files etc, we could no use the lstat or mtimes of previous build in this case.
• hashing of any kind (unless maybe permissable from the user) would cause our mem fs cache to continue to grow
• loader caching: the operation of one instance of a loader is likely comparable to a task in make is performing, with this in mind, we have already created a loader that will help users cache expensive loader operations.

I'm sure the list goes on. Regardless we are always looking for individuals to help performance tune all of the facets of our ecosystem so if you are interested in doing a more comparable, and far less hyperbolic analysis, then we are more than willing to lend a helping hand guiding you through the architecture

[Munter]

Make has its dependency graph defined in the makefile itself, which is what allows it to reason about work needing to be done. Webpack does not have this. It has entry points and must walk the dependency graph to it's full extent before it can start reasoning about the current state and compare to the previous one. These are radically different approaches, which will always have webpack come out slower. But I invite you to roll out your dependency graph into a flat make file yourself and see if you come up with a faster, useful and maintainable tool kit. Having worked for years in similar tooling I doubt the results will compete with make.

As for comparing start up time of a native binary with a script that depends on an entire node runtime to boot up, this feels out of scope for webpack specifically. You might want to open that part of the issue in the node project

[Munter]

Make has its dependency graph defined in the makefile itself, which is what allows it to reason about work needing to be done. Webpack does not have this. It has entry points and must walk the dependency graph to it's full extent before it can start reasoning about the current state and compare to the previous one. These are radically different approaches, which will always have webpack come out slower. But I invite you to roll out your dependency graph into a flat make file yourself and see if you come up with a faster, useful and maintainable tool kit. Having worked for years in similar tooling I doubt the results will compete with make.

As for comparing start up time of a native binary with a script that depends on an entire node runtime to boot up, this feels out of scope for webpack specifically. You might want to open that part of the issue in the node project