Webpack incremental builds are slow #616

Open
quicksnap opened this Issue Nov 25, 2015 · 120 comments

Projects

None yet
@quicksnap
Collaborator

On my machine, incremental builds in dev mode take anywhere from 1.5s - 5s.

I'm going to spend some time toying with webpack to see what's up, but I'd appreciate anyone else's ideas on what's going wrong.

@quicksnap
Collaborator

I'm able to drop the rebuild time significantly without bootstrap/font-awesome loaders in the main entry. Going to look into splitting it into a separate chunk.

@trueter
Contributor
trueter commented Nov 26, 2015

Sounds great!

@wwwfreedom

I'm getting slow build time as well. Looking forward to your split set up.

@luankefei

I am the same. it happened looks like webpack-hot-middleware didn't work.

@bdefore
Collaborator
bdefore commented Nov 26, 2015

@quicksnap if you set your devtool to inline-eval-cheap-source-map in dev.config.js i've noticed around 75% speed improvement on rebuilds, 10% improvement on initial builds. there's some discussion around why source mapping is expensive in recent versions, and it looks like it might be due to css-loader's implementation of css modules. you may also want to upgrade node-sass to 3.4.x. looks like the example project still has 3.3.3 which @phoenixmatrix identified as having poor performance: jtangelder/sass-loader#176

@cjhveal
cjhveal commented Nov 26, 2015

If I'm not mistaken, PR #614 upgraded node-sass to ^3.4.2.

@erikras
Owner
erikras commented Nov 27, 2015

Dupes #594

@quicksnap
Collaborator

Weird.. I thought I commented a reply to @bdefore:

Modifying or even removing the devtool doesn't do much for me. In my case, it's nearly all time due to the bootstrap-loader plugin. When I remove that, my rebuild time drops from ~4000ms to <500ms.

It would be nice to understand exactly what is going on. I tried digging into the plugin itself, but any modifications I made would not yield positive results, other than causing it not to generate the SASS file which @imports all the bootstrap stuff.

I have a feeling it's not properly caching due to its usage of a pitch loader. I'm not sure though, and it's tough to get help or even form an educated question.

Personally, I may just remove bootstrap loader and use a static build of bootstrap. I'm not sure the best route for this project.

Would love it if someone could see why bootstrap-loader isn't caching on rebuilds. Or anything to drop rebuild times <1000ms

@quicksnap
Collaborator

Perhaps it's simply that bootstrap generates a ton of CSS, and css-loader chokes on that.

I will try bypassing css-loader for bootstrap and see what happens.

@Phoenixmatrix

I don't know much on bootstrap-loader, but saw this since i got tagged on it. Its possible what's below is irrelevant, but I'll pitch it in anyway. Though it sounds like its just the loader not caching, as as mentioned.

I dont know what boostrap loader does, but do keep in mind that SASS doesn't get incrementally compiled the same way JS does. If you require(...) SASS, it creates a css "entry point" so to speak, but if inside that SASS, you @import other stuff, that's all part of the same "sass build". Sass builds are not incremental, and all SASS within a single "entry point" will get recompiled every time.

node-sass <3.4 is slow, and css-loader after 0.14.5 is BRUTALLY slow (to the point of being unusable). Since all of your CSS may get recompiled if its all in the same CSS entry point, you can quickly end up in CSS builds that take longer than rebuilding your entire JS from scratch, even though you only added 1 character to 1 CSS file.

Note that splitting your CSS by requiring it from JS isn't always a good answer in SASS because of the lack of @import (reference) like LESS does...you may end up duplicating a lot of stuff if you're not careful.

tl;dr: CSS builds using the @import mechanic are not incremental and cause a full rebuild of that part. You probably want a static bootstrap build, and @import only variables and mixins as need be (but nothing else!)

@quicksnap
Collaborator

@Phoenixmatrix thanks. The problem with this setup is that whenever we change any of our JS files, it triggers an entire rebuild, including the CSS. I could not figure a way to tell webpack not to rebuild bootstrap-sass every time. It is doing so here: https://github.com/erikras/react-redux-universal-hot-example/blob/master/webpack/dev.config.js#L54

I tried splitting out into a different chunk, but no luck. My webpack chops are mediocre.

I'm tooling around with bootstrap-sass-loader code now, but I still am not entirely sure where the slowness is being generated.

@Phoenixmatrix

Yeah, i realized that afterward rereading your comment. Whoops! For sure upgrading to node 3.4 should speed that up since compiling bootstrap shouldn't take very long at all, but obviously that only fixes symptoms (did you know node-sass is like 2-3 times faster on Linux than MacOSX? fun stuff).

Will take a look at the bootstrap sass loader when I get the chance.

@quicksnap
Collaborator

Thanks for helping out! Appreciate it.

Be sure to change the devtool to something fast like inline-eval-cheap-source-map. I forgot to do that and it was screwing up my benchmarks.

@quicksnap
Collaborator

Ok! I just changed devtool to inline-eval-cheap-source-map, and then edited bootstrap-sass-loader to this: quicksnap/bootstrap-sass-loader@6e60e0f (switch out css-loader for raw-loader)

This significantly speeds things up for me.

@quicksnap
Collaborator

I think it still would be an even better improvement if we could get webpack to avoid rebuilding bootstrap unless the bootstrap config file changes.

@Phoenixmatrix

Yeah, so obviously that part is still broken. But also, yet another reason
css-loader needs to be fixed. Its so completely broken performance wise
right now, as per webpack/css-loader#124

In our codebase its unusable (css build takes minutes, so we have to use
raw-loader)

@quicksnap
Collaborator

I hope it is improved by the time our css grows. I really like CSS Modules...

@mmahalwy
mmahalwy commented Dec 1, 2015

@quicksnap seems like you're suggesting to avoid bootstrap-sass-loader? That seems to be the bottleneck?

@bdefore
Collaborator
bdefore commented Dec 1, 2015

i had a chat with the shakacode folks this morning and discovered that bootstrap-sass-loader is being rewritten and will soon be deprecated. the rewritten version is here, though it hasn't been touched in a couple weeks: https://github.com/shakacode/bootstrap-loader/tree/alex/bootstrap-4

@quicksnap
Collaborator

@mmahalwy I'm not recommending avoiding it--I'll try to issue a PR to bootstrap-sass-loader soon. The change is minor. I don't see any problems using raw-loader over css-loader in bootstrap-sass-loader.

@Phoenixmatrix

Raw loader will not rewrite image and font URLs if you hash them. I don't
know if it affects bootstrap in any way.

On Tue, Dec 1, 2015, 1:24 PM Dan Schuman notifications@github.com wrote:

@mmahalwy https://github.com/mmahalwy I'm not recommending avoiding
it--I'll try to issue a PR to bootstrap-sass-loader soon. The change is
minor. I don't see any problems using raw-loader over css-loader in
bootstrap-sass-loader.


Reply to this email directly or view it on GitHub
#616 (comment)
.

-Francois Ward

@quicksnap
Collaborator

@Phoenixmatrix good point. I'll check on that, but I have a feeling it could cause issues now.

@quicksnap
Collaborator

I'll also look into passing params to css-loader to disable some things, or to downgrade css-loader in that repository.

@quicksnap
Collaborator

Or, having css-loader be a peer dependency.

@trueter
Contributor
trueter commented Dec 2, 2015

@quicksnap: I tried splitting out into a different chunk, but no luck. My webpack chops are mediocre.

What was the reason you gave up on this route? Having bootstrap outside the main chunk might circumvent the recompile issue.

@quicksnap
Collaborator

@trueter yes--I couldn't figure how to prevent bootstra-sass-loader from being triggered every rebuild. I did get it into a separate chunk successfully, but it would always rebuild all chunks AFAIK (it would say 'emitted' next to the vendor chunk I created).

@mmahalwy
mmahalwy commented Dec 4, 2015

@quicksnap any luck on this so far?

@quicksnap
Collaborator

@mmahalwy I haven't messed with adjusting bootstrap-sass-loader's properties yet. I'm also not sure if bootstrap-sass has any image URLs in it, so it may actually be good with raw-loader, which would be hella faster.

I'm working on stuff right now that doesn't involve dev mode, but next time I get hit with a 4sec rebuild I'll probably dig in.

@sars
Contributor
sars commented Dec 12, 2015

@quicksnap what's your build time now?
I've just launched this example and it takes 3.5-5 sec for me for each build...
it really slows down development..

Did you figure out how to speed it up?

@trueter
Contributor
trueter commented Dec 15, 2015

@quicksnap could you create a pr with your code split work in progress? stumbled across this: http://tech.trivago.com/2015/12/15/parallel-webpack/

@quicksnap
Collaborator

@sars @trueter Here's what will speed things up for me: https://github.com/erikras/react-redux-universal-hot-example/compare/master...quicksnap:foobar?expand=1

However, this could break things with bootstrap: raw-loader will not extract url() paths, and I'm unsure if bootstrap utilizes this. I do know they have some gylphicon stuff in there, so, it very well may cause issues.

@quicksnap
Collaborator

Whoops--you'll also need to npm i --save raw-loader in order for that to work.

@stevoland
Collaborator

I've just updated a project with a similar Webpack config (CSS Modules, node-sass etc) from Babel 5 to 6 (with es2015-loose). Incremental builds went from a horrible 4-6sec to a snappy 1-2sec. Might be worth looking into.

@trueter
Contributor
trueter commented Jan 20, 2016

Does it include bootstrap? Is it a public repo you can share?

@ozooner
ozooner commented Jan 20, 2016

He said it's a project with similar configuration...
There's an ongoing babel 6 upgrade issue #488

@isaacl
isaacl commented Feb 20, 2016

FWIW, disabling eslint-loader cut our build time by about 2x. eslint is just really slow, and the semantics of the loader api make it difficult to run eslint efficiently. I wrote a script (py | bash)** as an alternative but for an example app it should be packaged, etc. Some discussion here: MoOx/eslint-loader#82.

** not thoroughly tested!

@quicksnap
Collaborator

That's good to know!

@vjpr
vjpr commented Feb 25, 2016

If you have require('colors') anywhere in your codebase or any of your dependencies, and also use bootstrap-loader, check out the crazy bug I found here: reworkcss/css#88

My incremental build went from 14s to 7s.

For now just modify node_modules/resolve-url-loaders/node_modules/rework/node_modules/css/lib/parse/index.js directly to see if if helps.

@vjpr
vjpr commented Feb 25, 2016

I have been trying to work this out for my codebase (40s build / 10s incremental) and the issue it seems is a lack of easily accessible logging.

In Compilation.js, Compile#addModuleDependencies is called recursively and takes up the majority of the incremental build, while the build-module event is not firing. So my issue seems to be that it tries to build the dependency tree everytime without caching it, which is slow. I have asked a question here: webpack/webpack#2102 to confirm this.

Some ideas if dependency resolution trees are not cacheable:

  • Don't make webpack look inside every file in node_modules for dependencies. This can be done using the dist builds which most modules have. This makes debugging more difficult though :(. This can be done using noParse.
  • Identify which modules are expensive to resolve dependencies, and don't parse them. It appears that dependency resolution is asynchronous/parallel making it hard to isolate a single module resolution time (or is async used to do it in some order). If we could run this synchronously, we could easily see which modules take a long time to parse.
  • DLLCachePlugin - I'm very hopeful about this...I think it could solve my problem completely.
@trueter
Contributor
trueter commented Feb 25, 2016

Thanks for looking into that @vjpr! I tried commenting out the for in loop in parse/index but my incremental build time stayed around ~2000 ms using inline-eval-cheap-source-map on a 2014 MBPr. Removing bootstrap brings it down to ~750 so I feel most is to gain from tackling that.

@isaacl
isaacl commented Feb 25, 2016

@vjpr can you share the commands you used to profile webpack? are you looking at the output of webpack --profile --json? The screenshot looks like a v8 profile loaded in chrome

@vjpr
vjpr commented Feb 25, 2016

@isaacl At the moment I am using console.time, console.timeEnd statements manually added to webpack/lib/Compilation.js, to help me understand what is actually happening in webpack.

I also have used node-inspector to check flamecharts too which is where I found the colors issue.

@vjpr
vjpr commented Feb 26, 2016

After using the DllPlugin to compile my vendor modules separately, my incremental build is down from 10s to 3s, and main bundle build from 40s to 16s. Woohoo!

Code changes below. When I build my vendor bundle I set MAKE_DLL...

NOTE: I am using https://www.npmjs.com/package/webpack-configurator to manage my config if it doesn't make sense.

  const buildPath = path.join(opts.helpers.CWD, 'build')
  if (process.env.MAKE_DLL) {
    config.plugin('DllPlugin', webpack.DllPlugin, [{
      path: path.join(buildPath, '[name]-manifest.json'), // TODO(vjpr): Replace with build dir.
      name: '[name]_library',
      type: undefined,
      context: undefined,
    }])
  } else {
    config.plugin('DllReferencePlugin', webpack.DllReferencePlugin, [{
      //scope: undefined,
      //context: buildPath,
      context: cwd(),
      manifest: require(path.join(buildPath, 'vendor-manifest.json')),
      name: undefined, // Taken from `manifest` file.
      sourceType: 'var',
      type: undefined, // require
      scope: undefined,
      content: undefined,
    }])
  }

  if (process.env.MAKE_DLL)
    config.merge({
      entry: {vendor}
    })


  config.merge({output: {filename: !process.env.MAKE_DLL ? '[name].bundle.js' : '[name].dll.js',})

Also, watch out for this bug in DllPlugin: webpack/webpack#2106

@vjpr
vjpr commented Feb 26, 2016

Also, when you are adding to the vendor entry point, if you use a slash in the request parameter (e.g. require('lodash/merge')) this will bring in all of lodash to your main entry point.

You must add lodash/merge to your vendor entry point to avoid this.

Use --profile and --stats and search your console for ~/ to see which node_modules that are being pulled into your main entry point.

UPDATE: This helped me get incremental down from 3s to 1.5s, and normal build from 16s to 5s including bootstrap-loader.

UPDATE: I let my MacBook Pro Mid-2012 sleep overnight, swapped from Dell 30" monitor to Dell U2515h and now it is ~800ms. Not sure if it is related.

Web development is fun again!

NOTE: Waiting on these two bugs:

UPDATE:

  • If using OSX, if you use the Photos app, cloudd will try to upload all your photos and will slow your incremental builds down. This added 1s to my incrementals. Go to Photos > Preferences > Disable syncing for one day, or just disable it altogether.
  • iCloud is slow too. You may see the bird process using your CPU. Disable iCloud document syncing in the preferences pane.
  • Disabling iStats Menus allowed me to get from 700ms to 600ms.
  • Also fdbserver. Use launchctl to unload this, nothing will break, and it won't chew through cpu constantly.
@trueter
Contributor
trueter commented Mar 8, 2016

@vjpr Thanks a lot of taking the time to research this! I'm not able to get it working locally though:

      entry: {vendor}

What is the content of your vendor variable here?
I'm sure many people here would be eternally grateful if you could PR this.

@landabaso

@trueter it depends. For example, something like this:
var vendor = ["react", "react-dom"];

Put there anything from your node_modules you want to be bundled apart from your app.

@trueter
Contributor
trueter commented Mar 10, 2016

I was able to create the dll and manifest properly but now I'm stuck with this error message coming up when the application starts. The mentioned .scss file seems to be just a random file loaded early on. When I remove that require it fails at the next one.

[0] 2e806729f506f9eebf83b7fd00853881.png  27.4 kB          [emitted]  
[0]                           app.dll.js  4.43 MB       0  [emitted]  app
[0] /Users/yyyyy/xxxxx/xxxxx-react/src/components/Cover/Cover.scss.webpack-module:2
[0] exports = module.exports = require("./../../../node_modules/css-loader/lib/css-base.js")();
[0]                                                                                         ^
[0] 
[0] TypeError: require(...) is not a function
[0]     at Object.<anonymous> (/Users/yyyyy/xxxxx/xxxxx-react/src/components/Cover/Cover.scss.webpack-module:2:89)
[0]     at Module._compile (module.js:413:34)
[0]     at Object._module3.default._extensions.(anonymous function) [as .webpack-module] (/Users/yyyyy/xxxxx/xxxxx-react/node_modules/require-hacker/babel-transpiled-modules/require hacker.js:264:11)
[0]     at Module.load (module.js:357:32)
[0]     at Function.Module._load (module.js:314:12)
[0]     at Module.require (module.js:367:17)
[0]     at require (internal/module.js:16:19)
[0]     at populate_assets (/Users/yyyyy/xxxxx/xxxxx-react/node_modules/webpack-isomorphic-tools/babel-transpiled-modules/plugin/write assets.js:375:31)
[0]     at Object.write_assets [as default] (/Users/yyyyy/xxxxx/xxxxx-react/node_modules/webpack-isomorphic-tools/babel-transpiled-modules/plugin/write assets.js:63:2)
[0]     at Compiler.<anonymous> (/Users/yyyyy/xxxxx/xxxxx-react/node_modules/webpack-isomorphic-tools/babel-transpiled-modules/plugin/plugin.js:175:27)
@trueter
Contributor
trueter commented Mar 11, 2016

While looking for solutions on the webpack repo I found HappyPack, which can be used in combination with DLLs and cut our compile times in half just setting it up for the babel loader so far.

@isaacl
isaacl commented Mar 12, 2016

This seems like an issue with webpack itself -- does it cache incremental artifacts for webpack and webpack-dev-server builds? If so, why is it rebuilding node_modules stuff?

@trueter
Contributor
trueter commented Mar 21, 2016

I don't think that it's actually rebuilding node_modules stuff here. To me it looks like the issue is related to webpack-isomorphic-tools configuration for styles. @vjpr, do you use that plugin in your setup?

@jakubrohleder

@vjpr Can you share configuration part that handles bootstrap-loader?

I observed significant performance gains for initial load by using Happypack for my jsx files, while incremental changes stay around ~5s and most of this time is consumed by bootstrap-loader. Unfortunately whenever I try to make it work with Happypack or DLL the style are not loading.

@vjpr
vjpr commented Mar 22, 2016

I'm just adding bootstrap-loader to my vendor entry point. Nothing special.

@trueter
Contributor
trueter commented Mar 22, 2016

@vjpr Are you using css modules with local styles? If so, is your webpack-isomorphic-tools configuration different from this repo's ?
@jakubrohleder Are you getting similar errors?

@isaacl
isaacl commented Mar 23, 2016

@trueter the build is absolutely rebuilding node_modules stuff -- that's why DllPlugin hacks speed up the build. Also: incremental build speeds are poor even without webpack-isomorphic-tools -- why do you think that is the problem?

@jakubrohleder

@trueter I haven't got any error, just style were not displaying, I guess i forgot to link something. I had no time to fix it, but I'll be researching this issue more in the near feature.

@trueter
Contributor
trueter commented Mar 23, 2016

@isaacl Not sure i'm following. Sure, the webpack DLLPlugin is bundling modules together. But the error does not appear during compilation but when trying to load the application. The stack trace hints to the fact the dynamic require of a .scss-module does fail. Now the reasons I could see for this are that it either wasn't bundled properly in the first place or some hook is causing issues. Any way, since the module appears to live in memory (Cover.scss.webpack-module:2:89), I do not know how to dig into this further.
@jakubrohleder Alright, cool.

I would also be thankful if anyone came across useful webpack documentation regarding DLLs.

@trueter trueter referenced this issue in halt-hammerzeit/webpack-isomorphic-tools Mar 26, 2016
Closed

Compatibility with Webpack DLL Plugins? #62

@adailey14

@trueter @vjpr Just wanted to add another data point for anyone reading this.

I was able to get the DLL setup above working after some struggles. I packaged everything that was being required from node_modules into a vendor dll, and unfortunately this only reduced my initial build time for the non-node-modules code from ~40s to ~37s in my project.

What I conclude after some attempts at profiling is that the majority of the ~40s build time is being spent in the babel loader for my own js files. I found using happypack I was able to drop from ~40s to ~13s. I wonder if the folks seeing the big performance gains using the DLL setup are not ignoring node_modules in their webpack config like this:

    loaders: [
      { test: /\.js$/, exclude: /node_modules/, loaders: [strip.loader('debug'), 'babel']},
    ]

With that in place, I believe the remaining culprit for slow build times is just babel running on your own project files (my project has ~350 files). The happypack parallelization seems to help a lot. Happypack also seems to do some additional caching, though I'm not clear how that works or how much it helps.

Also sorry @trueter I'm not using css modules, so no insight into that error.

@ianks
ianks commented Apr 7, 2016

I saw my main perf gains from:

  1. Running typescript compiler as its own process using tsc --watch, then having webpack eat up the js files.
  2. Using cheap-module-source-map as my dev tool.

DLL did not significantly help my rebuild times.

@trueter
Contributor
trueter commented Apr 7, 2016

@adailey14 I learned the babel loader has a cacheDirectory query string flag that you can set to enable caching babel rebuilds. In a comparison caching app js files via happypack or caching them via babel cacheDirectory, the babel cache came out slightly ahead - you might want to give this a try.
By setting up a babel-specific happy pack loader you can set cache: false to leave caching to the babel cacheDirectory and still gain from the parallelization improvements for the initial build.

@amireh
amireh commented Apr 10, 2016

@jakubrohleder did you try HappyPack v2 for the styles? Previously, none of style-loader, css-loader, or sass-loader were supported. Personally, we don't use webpack for compiling styles and instead stick to vanilla node-sass or less (behind grunt or gulp) and it's been working fine...

@trueter curious how babel-loader's caching is faster for you than happypack's - can you share any benchmarks?

FWIW, I got HappyPack to work against this project and I think I'll submit a PR in case you guys want to integrate it. The gain wasn't that big as the number of modules is fairly small here; the initial build time went down from 11s to ~6.7s for me, and incremental ones are still at ~2.5s. However, the more modules you end up having, the bigger the gain.

@tearsofphoenix

@trueter
Could you post your webpack/dev.config.js with DllPlugin support please? Have try hours but con't make it work.
ps: Have read your slides:
http://www.slideshare.net/trueter/how-to-make-your-webpack-builds-10x-faster

@trueter
Contributor
trueter commented Apr 12, 2016

@amireh I didn't properly benchmark it or spend of lot of time researching it. Were you comparing the caches too and got different results? In this case I can go back and check again.

@tearsofphoenix There you go. Notice the first examples in the slides don't include the WebpackIsomorphicToolsPlugin.

@tearsofphoenix

Thanks! It works. But for me, happypack make building time decrease to 1 / 10, trumped the effect of DLLPlugin.

@trueter
Contributor
trueter commented Apr 15, 2016

You might want to check you configuration then, I believe with DLLPlugin you should be able to get below 1 second.

@amireh
amireh commented Apr 15, 2016

It's a good idea to use both DLLs and happypack. I use the DLLs for vendor libraries and local modules that do not change frequently.

It's a little sad webpack wasn't designed with concurrency in mind (and performance in general, how about an actual serializable, inter-process cache, for example? it's only somewhat okay at incremental builds, but takes forever to get to that phase...)

@FrendEr
FrendEr commented Apr 21, 2016

@vjpr have you used the code with the webpack-dev-server and if got some issues?

const buildPath = path.join(opts.helpers.CWD, 'build')
  if (process.env.MAKE_DLL) {
    config.plugin('DllPlugin', webpack.DllPlugin, [{
      path: path.join(buildPath, '[name]-manifest.json'), // TODO(vjpr): Replace with build dir.
      name: '[name]_library',
      type: undefined,
      context: undefined,
    }])
  } else {
    config.plugin('DllReferencePlugin', webpack.DllReferencePlugin, [{
      //scope: undefined,
      //context: buildPath,
      context: cwd(),
      manifest: require(path.join(buildPath, 'vendor-manifest.json')),
      name: undefined, // Taken from `manifest` file.
      sourceType: 'var',
      type: undefined, // require
      scope: undefined,
      content: undefined,
    }])
  }

  if (process.env.MAKE_DLL)
    config.merge({
      entry: {vendor}
    })


  config.merge({output: {filename: !process.env.MAKE_DLL ? '[name].bundle.js' : '[name].dll.js',})
@ianks
ianks commented Apr 21, 2016

I got my DLL working. At first I thought it did not increase rebuild times; however, I realized that it was not using the DLL, and just working anyway. The reason it was not using the DLL was because my context key was incorrect in my DLLReferencePlugin. For me, it needed to be set to the root of my project.

@FrendEr
FrendEr commented Apr 21, 2016

@ianks Do you use webpack-dev-server with DLL? I got some issue with it, I think I can get some help from you

@ianks
ianks commented Apr 21, 2016 edited

Yes I do. The main thing is you need a way to serve the DLL. webpack-dev-server exposes a setup configuration where you can do this. Then you have to put the DLL before your app bundle in your index.html. Here is how I do it:

  const devServerConfig = {
    // ...
    setup: app => app.use('/dll', express.static(path.resolve(__dirname, '..', '.tmp', 'dll'))),
    // ...
  };
@quicksnap
Collaborator

Just to chime in, I'm still not too happy with the rebuild times on my projects.

I'm hesitant to use Happypack/DLLs. I really don't want to add more complication to the build process. I also am using CSS local modules, so I can't downgrade css-loader =_(

Using the cacheDirectory option didn't seem to help out at all for me. Admittedly, I haven't put a ton of time into diagnosing the speed issues since many months ago.

Thanks for everyone digging into this and finding different solutions!

@trueter
Contributor
trueter commented Apr 21, 2016

@quicksnap you can use css modules with 0.14.5, just change the "modules" query var to "module". I had that in my slides that were posted earlier- check them out :)

@FrendEr
FrendEr commented Apr 21, 2016 edited

@ianks I use the webpack-dev-server CLI directly, so I'm wondering whether it's the reason.

When run npm run dev and view on browser with http://localhost:8808/webpack-dev-server/index.html, Console will throw the error...
Maybe you could help me with the code ? Thanks a lot.

@ianks
ianks commented Apr 21, 2016

you have to serve up the DLL bundle, which cant be achieved from the webpack-dev-server AFAIK @FrendEr. just make a script which uses the webpack-dev-server lib.

@quicksnap
Collaborator

@trueter thanks for the info.. will give that a shot soon!

@halt-hammerzeit
Contributor
halt-hammerzeit commented May 7, 2016 edited

When I was reading this issue the first time I thought webpack rebuilding times were not a bother for me.
But as my pet project grew bigger these times became a considerable factor in reducing productivity.
Today I tried "that DLL thing" and seems that it reduced my rebuilding times a bit: from > 2 sec to 1 sec.
Here's the commit with the relevant changes in case someone requires a guidance:
halt-hammerzeit/webapp@86e5722

I can say that 1 sec is around tolerable.
Still I think this whole thing is unacceptable and all this machinery should work lightning fast.

@jaraquistain jaraquistain referenced this issue in makeomatic/redux-connect May 7, 2016
Closed

support for defer: true #3

@tearsofphoenix

HMR takes almost 10+ seconds, I found each ES6 import syntax in src/server.js takes 2 ~ 3 seconds, anyone has ideas ?

@halt-hammerzeit
Contributor

My server takes about 4 secs to start because of importing all the client
side code too

On Monday, May 9, 2016, tearsofphoenix notifications@github.com wrote:

HMR takes almost 10+ seconds, I found each ES6 import syntax in
src/server.js takes 2 ~ 3 seconds, anyone has ideas ?


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#616 (comment)

@hanjukim
hanjukim commented May 9, 2016 edited

@tearsofphoenix
I have added below code into server.babel.js to investigate.

  config.ignore = function ignoreFn(filename) {
    if (/node_modules/.test(filename)) return true;
    console.log(filename);
    return false;
  }

As you will see, server restarting needs time as I add components more and more, so I came to resolution that I turn off SSR for development mode, turn off piping in bin/server.js, and depend on webpack HMR only. I am also using https://github.com/amireh/happypack

@halt-hammerzeit
Contributor

@hanjukim that's a good enough idea: SSR can be disabled and you get lightning fast reload time (and you don't reload the server when you edit your components at all).
An alternative solution would be to extract webpage rendering server into a separate app like I do in my project with page-server:
https://github.com/halt-hammerzeit/webapp/tree/master/code
In this scenario page-server takes about 4 secs to start but it doesn't matter when it isn't used at all if you aren't refreshing the webpage.

@trueter
Contributor
trueter commented May 9, 2016 edited

bin/server.js restart time also became a frustrating factor for me. HMR unfortunately does not work reliably with stateless function components yet. Putting my hopes on React Hot Loader 3. Changing the babel language level to node6 on the server could also help with import time. I haven't found the time yet to properly explore either option, but I'm curious if you do.

@halt-hammerzeit
Contributor
halt-hammerzeit commented May 9, 2016 edited

@trueter Will also try "React Hot Loader 3" when it ships (currently no server side rendering).

babel-preset-node6 turned out to be a good thing, shove about 10%-15% off the loading times (or maybe it's just my perception).

{
    "presets":
    [
        "react", 
        "es2015", 
        "stage-0"
    ],

    "plugins":
    [
        "transform-runtime",
        "add-module-exports",
        "transform-decorators-legacy",
        "transform-react-display-name"
    ],

    "env":
    {
        "server":
        {
            "presets":
            [
                "react", 
                "node6", 
                "stage-0"
            ],

            "plugins":
            [
                "transform-runtime",
                "add-module-exports",
                "transform-decorators-legacy",
                "transform-react-display-name"
            ]
        }
    }
}

  "betterScripts": {
    "development-page-server": {
      "command": "nodemon ./code/page-server/entry.js --watch ./code/page-server --watch ./code/client --watch ./code/common",
      "env": {
        "NODE_ENV": "development",
        "BABEL_ENV": "server"
      }
    },
    ...
@slavab89

@trueter I've tried using the config that you've posted before but i dont quite understand how to make it work properly.
I've create a new npm script that runs with MAKE_DLL=1 and it does create a vendor.dll and an app.dll BUT while creating them i have some error regarding the fact that it cannot find anything under src/containers and src/redux directory. I believe that its due to the fact that there is no 'src' folder under resolve.moduleDirectories. When i put it back there it seems to build but i'm not quite sure why its not it your config. Also when trying to run it with the .src file in there, webpac-isomorphic-tools fails to find ./src/theme/bootstrap.config.js

Also, just wanted to make sure here. When using this approach, i need to first run a webpack build that creates the dll using MAKE_DLL=1 and after that run webpack dev server with MAKE_DLL=0 so that it will just bootstrap the previously created dll and watch only the files under .src. correct?

@halt-hammerzeit
Contributor

Also, just wanted to make sure here. When using this approach, i need to first run a webpack build that creates the dll using MAKE_DLL=1

yes

and after that run webpack dev server with MAKE_DLL=0

yes

so that it will just bootstrap the previously created dll and watch only the files under .src. correct?

no, it won't "bootstrap" anything, you'll have to add a separate <script src="vendor.dll.js"/> on your page

@trueter
Contributor
trueter commented May 11, 2016 edited

When i put it back there it seems to build but i'm not quite sure why its not it your config

I removed it on purpose because it adds complexity for webpack to resolve from where your module is required. You can add src back to your moduleDirectories for convenience, but since this thread is all about reducing webpack build times, check out webpack aliases instead.

Also when trying to run it with the .src file in there, webpack-isomorphic-tools fails to find ./src/theme/bootstrap.config.js

I am configuring my bootstrap-loader via a .bootstraprc file. Guess your issue is just about updating a path.

When using this approach, i need to first run a webpack build that creates the dll using MAKE_DLL=1 and after that run webpack dev server with MAKE_DLL=0 so that it will just bootstrap the previously created dll and watch only the files under .src. correct?

Yes. I'm outlining what happens exactly in my slides posted earlier.

@alecperkey
alecperkey commented May 25, 2016 edited

If anyone using this boilerplate has already modified their config according to the advice in this thread (happypack, dll bundles) would be awesome to see a gist.

Basically I'm stuck converting

from this:
https://github.com/erikras/react-redux-universal-hot-example/blob/master/webpack/dev.config.js

to this:
https://gist.github.com/trueter/0e861403e59a9e27a476f3ad7ada1a89

I have this error:

[0] Error: Cannot find module '../static/dist/vendor-manifest.json'
[0] at Object.<anonymous> (C:\Users\User\repos\react-client\webpack\dev.config.js:166:19)

which this snippet of the gist

new webpack.DllReferencePlugin({ context : path.join( __dirname, '..' ), manifest: require('../static/dist/vendor-manifest.json') }),

@trueter

  1. Where does the vendor-manifest come from?
  2. What is the 'gotcha' with the isomorphic tools exactly?
    I assume something to do with this part of the stack trace... but my understanding is unclear.

[0] at Function._module3.default._resolveFilename (C:\Users\User\repos\react-client\node_modules\webpack-isomorphic-tools\node_modules\require-hacker\babel-transpiled-modules\require hacker.js:403:34)

~ Looking forward to hearing your insight!

@halt-hammerzeit
Contributor

@alecperkey I'd say we answer your questions here and when you make it working you submit a PR into this repo.
Fair enough.

As for your questions, dist/vendor-manifest.json is created in the same folder where the DLL is output.
And you tell it yourself where to output the dll
https://github.com/halt-hammerzeit/webapp/blob/master/webpack/dll.js#L64-L68

I had no "gotchas" with isomorphic tools in my setup.

@slavab89

@alecperkey You are missing a run command. It cannot find the vendor dll because you havent created it. This is why in the example that you've followed it has 2 tasks. 1 for creating the dll and the other for actually using it.
I'm not sure my configuration is valid therefore i'm not opening a PR yet but this is my section of the creation.
It does create a vendor.dll for me and seems to be using it (i hope 😃 ) but it does not pack the bootstrap (not sure how to do it yet) and the initial build time is still around 10s and later its about 2-4 seconds
You also need to create an additional task that runs 2 builds, something like

"build-watch-client": "npm-run-all client:prepare watch-client",
"watch-client": "better-npm-run watch-client",
"client:prepare": "MAKE_DLL=1 webpack --colors --display-error-details --config webpack/dev.config.js"

Again, i might be mistaken with the config so someone can correct me or add what i'm missing (would like to know how to include the bootstrap in the vendor as well)

if(process.env.MAKE_DLL) {
  module.exports.entry = {
    vendor: vendorLibraries(),
  }
  module.exports.output = {
    path: assetsPath,
    filename: '[name].dll.js',
    library: '[name]_library',
    publicPath: 'http://' + host + ':' + port + '/dist/'
  }
  module.exports.plugins.unshift(
    new webpack.DllPlugin({
      path: path.join(assetsPath, '[name]-manifest.json'),
      name: '[name]_library'
    })
  )
} else {
  module.exports.entry = {
    'main': [
      'webpack-hot-middleware/client?path=http://' + host + ':' + port + '/__webpack_hmr',
      'bootstrap-sass!./src/theme/bootstrap.config.js',
      'font-awesome-webpack!./src/theme/font-awesome.config.js',
      './src/client.js'
    ]
  }
  module.exports.output = {
    path         : assetsPath,
    filename     : 'main.js',
    publicPath   : 'http://' + host + ':' + port + '/dist/'
  }

  module.exports.plugins.unshift(
    new webpack.DllReferencePlugin({
      context : path.join( __dirname, '..' ),
      manifest: require('../static/dist/vendor-manifest.json')
    })
  )
}
@alecperkey
alecperkey commented May 25, 2016 edited

How does one choose which packages are appropriate for the vendorArray?
For example, I know React should be in there, but it seems wrong to just put paste all the package names from dependencies and devDependencies there.

Trying to update the RRUHE boilerplate... but I am getting an error when running the 'client-prepare' script even.

See fork here: alecperkey@5e77403

running better-npm-run in C:\Users\User\repos\react-redux-universal-hot-example
Executing script: client-prepare

to be executed: webpack --colors --display-error-details --config webpack/dev.config.js
net.js:617
    throw new TypeError('invalid data');
    ^
@alecperkey

This commit I've removed some devDependencies which probably dont need to be in the vendorArray.. but I'm unsure which other ones are unnecessary.

And the npm script 'client-prepare' I have no idea what 'invalid data' error is. If i run the webpack command directly, it works, but I need the better npm run for the MAKE_DLL=1 variable on windows.

@slavab89

@alecperkey Might be some windows and better npm thing that does not work together? Maybe that's not how you define a variable for the script to use?

Regarding the vendor array. Basically anything that you use on the client side can go there. For example stuff like express should not go there because that's sever side (i think it will also not let you and wont work) but anything client related can go there. I've just gone over all the import statements of the client files and took the packages

@alecperkey
alecperkey commented May 31, 2016 edited

@slavab89, thanks for the suggestion. It feels so close!

Not sure what the better npm run problem is... cause it works fine on a different repo for setting env variables this way. However, I've just made a 'quick hack' workaround for now.

I updated the vendor array with your method suggested above.
There is just one error remaining -- when generating the app DLL bundle. It is in the ERROR LOG.txt

[TypeError: The plugin ["react-transform",{"transforms":[{"transform":"react-transform-hmr","imports":["react"],"locals":["module"]}]}] didn't export a Plugin instance]

If anyone can help, see commit: alecperkey@2ecde40

@jaraquistain

I wanted to use this thread as a starting point for improving my webpack config and it occurred to me that it could be useful if people who have spent a significant time on this subject maybe posted a gist of their webpack config for reference. Anyone willing to throw one up?

@ianks
ianks commented Jun 2, 2016 edited

Here is my dll.config.js file:

'use strict';

const _ = require('lodash');
const baseConfig = require('./base');
const path = require('path');
const webpack = require('webpack');

const config = _.merge(baseConfig, {
  entry: {
    vendor:  Object.keys(require(path.resolve(__dirname, '..', '..', 'package.json')).dependencies),
  },
  output: {
    path: path.resolve(__dirname, '..', '..', '.tmp', 'dll'),
    filename: '[name].dll.js',
    library: '[name]_[hash]'
  },
  cache: false,
  devtool: 'inline-source-map',
  plugins: [
    new webpack.ProvidePlugin({
      $: 'jquery',
      jQuery: 'jquery',
      jquery: 'jquery'
    }),
    new webpack.DefinePlugin({
      __DEV__: true,
      'process.env.NODE_ENV': '"development"'
    }),
    new webpack.DllPlugin({
      path: path.resolve(__dirname, '..', '..', '.tmp', 'dll','[name]-manifest.json'),
      name: "[name]_[hash]",
    }),
  ]
});

config.module.loaders = baseConfig.module.loaders.concat([
  {
    test: /\.css$/,
    loaders: ['style', 'css'],
  },
  {
    test: /\.scss$/,
    loaders: ['style', 'css?sourcemap', 'sass?sourcemap'],
  },
]);

Which is used by my dev.config.js

'use strict';

const config = _.assign(baseConfig, {
 ...
  cache: true,
  devtool: 'cheap-module-inline-source-map',
  plugins: [
    ...,
    new webpack.DllReferencePlugin({
      context: path.resolve(__dirname, '..', '..'),
      manifest: require(path.resolve(__dirname, '..', '..', '.tmp', 'dll', 'vendor-manifest.json')),
    }),
   ...
  ],
});
@halt-hammerzeit
Contributor

gosh, just make that PR, ppl

i can answer any question on this topic

@alecperkey

@halt-hammerzeit, I'd make the PR would like it working first.

Stuck on this error from my last comment: #616 (comment)

Its some problem with babel react-transform plugin 'not exporting a plugin instance'.
master...alecperkey:master

Any advice?

Looking into it I found https://github.com/gaearon/babel-plugin-react-transform is undergoing deprecation to be replaced by React Hot Loader 3.

@AndrewRayCode
Contributor
AndrewRayCode commented Jun 27, 2016 edited

I'm trying to use the DllPlugin, I think I have the Dll files building fine, but when I run my dev server I get:

[1] [webpack-isomorphic-tools] [error] asset not found: ./assets/images/earth-texture.jpg
...
[1] [webpack-isomorphic-tools] [error] asset not found: ./src/components/Columns/Flows.scss
...

It seems like it can't find lots (maybe all of?) my scss and image files to process. Here's my config.dll.js file.

In my webpack/dev.config.js file I include all of the manifests:

 plugins: [
    new webpack.DllReferencePlugin({ 
      context: process.cwd(),
      manifest: require(path.join(assetsPath, 'main-manifest.json'))
    }),
    new webpack.DllReferencePlugin({ 
      context: process.cwd(),
      manifest: require(path.join(assetsPath, 'vendor-manifest.json'))
    }),
    new webpack.DllReferencePlugin({ 
      context: process.cwd(),
      manifest: require(path.join(assetsPath, 'assets-manifest.json'))
    }),

and in main-manifest.json I can see all of the images and scss files listed that webpack-isomorphic-tools are telling me are missing. Do I need to tell isomorphic tools somehow about the manifest json files?

@halt-hammerzeit
Contributor

@DelvarWorld I don't think DLL is a place for assets. Conversely, I guess they shouldn't be there.
I don't know why are they there.
Maybe you could try to remove assets from there
https://gist.github.com/DelvarWorld/4294145ba1326b048cd1e17174220ddd#file-dll-config-js-L25

@AndrewRayCode
Contributor
AndrewRayCode commented Jun 27, 2016 edited

@halt-hammerzeit part of why I'm doing this is I'm including some very large json files (they are 3d model files) and i'm trying to cache those into separate entry points so they don't all get re-included in the main bundle. But they don't all live in assets.js, in fact none of the errors I'm getting come from that file, they're all from the app entry point ./src/client.js.

I notice the pattern in some places where the image path or css path is required inside the render function - should I be doing this? Right now I require them at the top level of each file like any other import.

@halt-hammerzeit
Contributor

@DelvarWorld Well, I don't know what's going on there but I don't think you can cache assets with webpack-isomorphic-tools (maybe with universal-webpack). If your assets aren't in webpack-stats.json then they won't be found. webpack-stats.json are generated by Webpack.

@amireh
amireh commented Jun 27, 2016

By the way, posting here since it's relevant to this discussion: there's a pending PR for DLL support (#1201) but it bases off the happypack PR #1090. With some effort, you could define an asset DLL (some README exists for defining more DLLs). There might be issues with webpack-isomorphic-tools though.

Perhaps you could try those patches out?

@AndrewRayCode
Contributor

@halt-hammerzeit Thanks for the help so far, I've discovered when I run my dll initial build the webpack-assets.json gets populated correctly, or at least it has all the image and scss paths in there. But when I run the default npm run dev it overwrites the contents of that file with:

{
  "javascript": {
    "main": "http://localhost:3001/dist/main-826b5781622f36dec637.js"
  },
  "styles": {},
  "assets": {
    "./~/font-awesome/fonts/fontawesome-webfont.eot": "http://localhost:3001/dist/404a525502f8e5ba7e93b9f02d9e83a9.eot"
  }
}

This is my only change to my wepback/dev.config.js file, other than removing CommonChunksPlugin

+    new webpack.DllReferencePlugin({
+      context: process.cwd(),
+      manifest: require(path.join(assetsPath, 'main-manifest.json'))
+    }),
+    new webpack.DllReferencePlugin({
+      context: process.cwd(),
+      manifest: require(path.join(assetsPath, 'vendor-manifest.json'))
+    }),
+    new webpack.DllReferencePlugin({
+      context: process.cwd(),
+      manifest: require(path.join(assetsPath, 'assets-manifest.json'))
+    }),

For my dev.config.js (not the DLL one), my entry point is simply:

  entry: {
    main: [
      'webpack-hot-middleware/client?path=http://' + host + ':' + port + '/__webpack_hmr',
      'font-awesome-webpack!./src/theme/font-awesome.config.js',
      './src/client.js'
    ],
  },
@halt-hammerzeit
Contributor

@DelvarWorld

Thanks for the help so far, I've discovered when I run my dll initial build the webpack-assets.json gets populated correctly, or at least it has all the image and scss paths in there. But when I run the default npm run dev it overwrites the contents of that file with.

Oh, maybe that's the issue - it overwrites the file.
You can try to disable webpack-isomorphic-tools plugin for the second pass so that it doesn't overwrite the correst assets file with the new empty one.

@AndrewRayCode
Contributor
AndrewRayCode commented Jun 27, 2016 edited

Ok cool removing webpack-isomorphic-tools from the main webpack config file preserves the manifest. Something else that's confusing is that now all of my assets can only be served from :3000 instead of :3001. I see this line in a linked config, where 1 is subtracted:

publicPath   : 'http://' + host + ':' + ( port - 1 ) + '/dist/'

But I'm not sure why this is needed? When my code starts it logs:

[0] ==> 🚧  Webpack development server listening on port 3001

But all files at :3001 error (they didn't error before the DllPlugin), and those paths only exist at :3000 now. Do you know why this is happening, and if it's an error or expected behavior?

@halt-hammerzeit
Contributor

The files should be available through webpack-dev-server (3001 in your case). If they aren't then it may seem that either their filenames are from another build and therefore are different or that the files aren't included in the (second?) build which could also be true since they are absent from the assets.

@AndrewRayCode
Contributor

@trueter did you run into this issue and do you have any insight as to why you did port - 1 in your config?

@AndrewRayCode
Contributor
AndrewRayCode commented Jun 28, 2016 edited

Ok, finally got somewhere with the DllPlugin. I'm going to document everything I know here, for now, because the information on this is scattered, hard to find and sometimes convoluted. There are still many unknowns in this process, and I will update this reply if any of the unknowns are clarified.

Warning: It's easy to do this incorrectly. You can actually double your build size by accidentally bundling your dependencies into the Dll bundle and your main bundle. Several people in this thread have said the DllPlugin gave them no speed up, but I wonder if it was just a configuration issue.

Warning: These steps are currently specific to this project, and more specifically to webpack-dev-middleware. However the DllPlugin can be used in any Webpack project.

Warning I haven't set up the final script tag to work in a production build (yet), but it should be straightforward.

TL;DR dll.config.js and dev.config.js and hard code <script src="http://localhost:3001/dist/app.js" charSet="UTF-8"/> into Html.js but you might want to read this anyway!

Reducing My Hot Reload Time From 15s to 4s With The DllPlugin

I still have more work to do on optimizing (next will try HappyPack), but this is a serious improvement already.

My Specific Problem

I have many large asset files that are being built into my bundle. I'm building a 3d game using React and the model files are in JSON, so I have many MB of 3d model data to load. Even though they're going through the file loader, they still get built into the main bundle. I'm also including large Javascript libraries, like Three.js.

Webpack Is Slow

Out of the box, Webpack isn't efficient. In fact, it's surprising it works the way it does. Webpack treats everything, both your source and your dependencies, as one big bundle. Even in development mode. What this means in practice is that if you change any source file, ALL Javascript files, including dependencies, are re-processed and re-bundled. You can see why this would be slow. I believe this happens even if you use the CommonChunks plugin. If you change your source file, even with CommonChunks, the common chunk will get re-written (I could be wrong but that's what the last section of this post suggests).

What Is The DllPlugin?

Really, if you think about it, we don't want Webpack to touch our third party dependencies more than once. That's where the oddly named DllPlugin comes in. I don't know why it reuses the Windows development "Dynamically Linked Library" terminology, but that's what it stands for. The basic workflow is this:

  1. Create a separate, one time build step that uses the DllPlugin
  2. Run this one-time build step before your dev server. It creates both a bundle file containing your third party code, and a "manifest" file, which maps require paths to built Webpack ids, so your app code (via Webpack) knows how to find them at runtime.
  3. Modify your webpack dev config to use the DllReferencePlugin to point to the manifest file(s) built in step 2.

Don't overlook the importance of caching this manifest file. Without the DllPlugin it's re-created every time for every code change, as in Webpack walks all your dependencies to build it. Caching it for third party node modules seems like a good idea to me!

Setting up the DllPlugin

You have two options here. You can either create an entirely separate Webpack config file for the Dll build, or you can modify your existing dev.config.js file to build the Dll files if you set an environment variable. The benefit of an environment variable switch is less code duplication, the downside is it's harder to read and makes the config file more confusing. The downside of a separate file is you'll probably duplicate some config options, violating the DRY principle. I don't think either option is ideal, so go with personal preference. For this tutorial, I'm going to create a separate dll.config.js file.

Creating The Dll Configuration File

This will be a modified copy of your dev.config.js file. We'll go over each thing you need to change.

I've put the my completed dll.config.js file online for you to reference.

Every step below is done in a copy of webpack/dev.config.js that I arbitrarily name webpack/dll.config.js

1) Add an entry for your main app, but name it something different. Also it must be an array, not a single string.

entry: {
  app_assets: [ './src/client.js' ],

According to this comment, if you don't include your main entry file, the main webpack-assets.json manifest file won't include your runtime static assets like CSS, JS, etc.

2) Add an array of the third party vendor code to go into the cached manifest and bundle:

entry: {
  app_assets: [ './src/client.js' ],
  vendor: [
    'react',
    'react-dom',
    ...
  ]

How did I come up with this list? Based on an earlier comment, I modified webpack-dev-server.js to set quiet: false and noInfo: false, started the dev server, then manually copied anything starting with ./~/ into the vendor: [] array. Well, most things. I excluded Webpack-looking things like the css-loader which appears in that list.

One gotcha here is if you try to include 'babel-runtime' (which is a huge dependency) in the array it will fail, complaining it can't find it. I don't know why, but change it to 'babel-runtime/core-js'.

By the way, when this is all over, you can set the server flags again (quiet: false and noInfo: false,) and see which files are now coming from your cached manifest. They will look like:

delegated ./node_modules/redux/lib/index.js from dll-reference vendor 42 bytes {0} [not cacheable]

3) Set your output section to exactly:

  output: {
    path: assetsPath,
    filename: '[name].dll.js',
    library: '[name]',
    publicPath: 'http://' + host + ':' + ( port - 1 ) + '/dist/'
  },

Through trial and error I found that library: '[name]', is required. If you want you can change the .dll.js to something, like .bundle.js, but keep [name].

4) Include the DllPlugin

    new webpack.DllPlugin({
      name: '[name]',
      path: path.join( assetsPath, '[name]-manifest.json' ),
    }),

You may have noticed that earlier comments on this thread include things like context: undefined and other config keys. You don't need them. Again you can change the -manifest.json string, but remember what you change it to because we need it.

5) Verify your plugins

Make sure you include webpackIsomorphicToolsPlugin.development(), in the plugins: [] list for your dll.config.js file.

Modifying Your Main dev.config.js file

My final dev.config.js file is online for you to compare against.

1) Remove everything from your entry config except your main entry point

  entry: {
    main: [
      'webpack-hot-middleware/client?path=http://' + host + ':' + port + '/__webpack_hmr',
      'font-awesome-webpack!./src/theme/font-awesome.config.js',
      './src/client.js'
    ],
  },

Standard stuff here.

2) Modify your output to only build one file with an exact name

  output: {
    path: assetsPath,
    filename: 'app.js',
    publicPath: 'http://' + host + ':' + port + '/dist/'
  },

We'll come back to this later, see Unknown 2 at the end of this post for why we don't use the good old [name]-[hash] style.

3) Add DllReferencePlugin(s) to your plugins list:

  plugins: [
    new webpack.DllReferencePlugin({ 
      context: path.join( __dirname, '../' ),
      manifest: require(path.join(assetsPath, 'vendor-manifest.json')),
    }),
    ....

Add one DllReferencePlugin call for each entry you specify in dll.config.js. If you only have one vendor entry (most common case) then you only need one plugin here. Keep in mind it must match the string pattern you specified in dll.config.js (in our case [name]-manifest.json).

Do they need to go at the beginning of the plugins list? I don't know. A previous comment used unshift to put the DllReferencePlugin at the beginning of the array. I'm not deviating.

4) Remove the webpackIsomorphicToolsPlugin and CommonsChunkPlugin

Do not include webpackIsomorphicToolsPlugin nor CommonsChunkPlugin in your plugins: [] list for webpack/dev.config.js. CommonChunks can't co-exist with DllPlugin. They are competing solutions, and one clobbers the other. For why we remove webpackIsomorphicToolsPlugin, see Unknown 2

Add your new bundles to Html.js

In this project's Html.js file, include your vendor script tag(s):

<script src={assets.javascript.vendor} charSet="UTF-8"/>

Now include your hard coded app.js file, which we created in the above dev.config.js file:

<script src="http://localhost:3001/dist/app.js" charSet="UTF-8"/>

Run the manifest step and then run your project!

Here's the command I use to create the cached manifest/bundle files. You only have to run this command once, and then only again if your dependencies change:

webpack --colors --progress --config webpack/dll.config.js

Then start your server like normal:

npm run dev

If everything went well, you should have faster hot reloading!

Unknowns

Unknown 1:

According to @sokra:

If you process the file with the file-loader, they should not be included in the bundle file, but emitted as separate files

However, this boilerplate project uses webpack-isomorphic-tools and webpack-dev-middleware instead of the regular dev server, and according to the middleware docs:

No files are written to disk, it handle the files in memory

Does using webpack-dev-middleware mean that large files are always re-created in memory on every hot reload? Would this be solved by using the regular dev server and just putting files on disk that aren't touched again on hot reload? I have no idea, but I made a ticket to ask.

Unknown 2:

In the above code we removed the isomorphic tools plugin from the main dev config, and hard coded the asset path. So here's the issue. If I include webpackIsomorphicToolsPlugin in my dev.config.js file, it overwrites the contents of webpack-assets.json, which was previously created by the Dll step. This means that all the cached manifest data is clobbered. I have no idea if this is supposed to work this way, and if you're not supposed to use it in the main dev server, but...

...The other problem here is that without webpackIsomorphicToolsPlugin running on your source code, assets.javascript.main won't exist, because it's not added to the manifest. I believe webpackIsomorphicToolsPlugin is reponsible for the addition. That's why I have to hard code the path. Also this means if you want to load it from the same port as the other assets (which are 3000) you'd have to manually expose that file by adding a server route.

That's it!

This is a first draft, and hopefully it gets some good input and feedback and is refined. I don't know if this is the solution, or even an ideal solution. I've just been hacking at this frustrating problem for 2 days straight and had to jump between many sources. The Webpack docs are no help, as expected. Hopefully this helps someone!

Further reading: Webpack Plugins we been keepin on the DLL and Optimizing Webpack Build Times

(Spam) if this helped you consider following me on Twitter

@halt-hammerzeit
Contributor

Good post

Does using webpack-dev-middleware mean that large files are always re-created in memory on every hot reload?

If your file is processed by webpack dev server (in any way) then it resides in RAM and is recompiled only on changes to it. I would suppose the DDLRefPlugin is intelligent enough to exclude files from the DLL from webpack dev server's list of files.

I have no idea if this is supposed to work this way

It is supposed to always overwrite webpack-assets.json on a subsequent run.

and if you're not supposed to use it in the main dev server, but...

You do it in the main dev server if you're not gonna put your assets into the DLL. Otherwise, I guess, you do it in the DLL compilation run (for whatever reasons you have).

without webpackIsomorphicToolsPlugin running on your source code, assets.javascript.main won't exist

Yes, assets.javascript.main is taken from webpack-assets.json, so if it's wrong or non-existent there then it will be the same when you run the app.

(you might also consider trying universal-webpack later)

@amireh
amireh commented Jun 28, 2016 edited

The solution @DelvarWorld outlined in great detail is close to the one proposed in PR (#1201), perhaps these should converge? The only differences as far as I can tell is that it doesn't disable CommonsChunk or isomorphic-tools (although it probably should) and it tracks babel-runtime modules in the vendor DLL.

Edit: sound less obnoxious - was just trying to help. 😁 Have a great week everyone!

@oyeanuj oyeanuj referenced this issue in mxstbr/react-boilerplate Jun 28, 2016
Merged

Implement Webpack DLL Plugin #495

6 of 6 tasks complete
@Jack-Barry
Jack-Barry commented Aug 4, 2016 edited

It's not all that elegant but what I've done to make hot updates snappy is to run a separate, concurrent Webpack process to bundle my Bootstrap, with a configuration that doesn't have hot updates enabled. In simple terms, the basic syntax is:

module.exports = [
    { config for regular bundle with hot updates enabled, css loader ignores bootstrap },
    { config for Bootstrap using raw loader without hot updates }
];

This way, it gets passed through the loaders on initial build, but isn't touched during incremental builds. It's also running concurrently on the initial build with my regular bundle which shaves a few seconds off at that point. Still pretty long build time initially, but it seems to help for incremental builds which are more aggravating when slow. I'd still like to speed up that initial build but that seems a lot more doable using something like this. I'm thinking of setting up a webpack-vendor.config.js so I can do something like run webpack on that as needed (just needs to build the files once on a new machine, or be re-run whenever dependencies are updated), and use webpack-dev-server with hot updates to serve up my actual app code, but I need sleep first.

I'm no pro so if this approach would bite me in the ass please let me know before I dig too deep.

@seeliang
seeliang commented Sep 30, 2016 edited

Thanks @trueter,
with the babel cacheDirectory set
my rebuild drops form 2.5s to 1.5s

@seeliang
seeliang commented Oct 5, 2016

Try cache : true,

Now 1.5s to build 0.6s to rebuild !!

module.exports = {
  cache: true,
  devtool: 'cheap-module-eval-source-map',
  entry: './src/scripts/index.js',
  externals: {

  },

  module: {
    loaders: [
      {
        loader: 'babel-loader',
        exclude: /node_modules/,
        query: {
          presets: ['es2015'],
          cacheDirectory: true
        }
      }
    ]
  },

  output: {

  },

  plugins: [
    new webpack.optimize.UglifyJsPlugin({ output: {comments: false}})
  ],

  resolve: {
    modulesDirectories: [
      'node_modules'
    ]
  }
}`
@echenley
echenley commented Oct 6, 2016

@seeliang cache: true is already enabled by default in watch mode https://webpack.github.io/docs/configuration.html#cache. I've noticed with Webpack 1.x the first rebuild takes longer than subsequent rebuilds, which can be a red herring when tweaking your build.

@seeliang
seeliang commented Oct 7, 2016 edited

@echenley thanks for the update.

I really not 100% sure why i need cache: true or how it works,
one post mentioned this set

and it did improve my build :)

@seeliang
seeliang commented Oct 12, 2016 edited

if you are using babel with lodash

try
https://github.com/lodash/babel-plugin-lodash

@nndung179

I am using Happypack & DLL plugin together. Currently, the performance is not so bad. It reduced from more than 15s to 4-5s when it rebuild

@vhpoet
vhpoet commented Oct 18, 2016 edited

has anyone tried webpack unsafe caching (another link)? This is also supposed to make incremental builds faster.

@bebraw
bebraw commented Dec 6, 2016

hard-source-webpack-plugin might come in handy here.

@nezed
nezed commented Dec 14, 2016 edited

Also you can easily improve performance by moving modules that you have no intention of changing into new chunks by specific/extra conditions (e.g. module path or module size) 🔥 🚀

Check the webpack-split-chunks

Example:

    plugins: [
        new ChunksPlugin({
            to: 'vendor',
            test: /node_modules/ // can be function | RegExp | Array[RegExp] 
        })
    ]
@oychao
oychao commented Jan 19, 2017

@bdefore Many thanks. You made my day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment