New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webpack DLL Plugin #702

Closed
mikecx opened this Issue Aug 25, 2017 · 15 comments

Comments

Projects
None yet
7 participants
@mikecx
Copy link

mikecx commented Aug 25, 2017

Has anyone been able to get the Webpack DLL Plugin to work with webpacker? I followed the guide here: http://engineering.invisionapp.com/post/optimizing-webpack/ and ended up with a DLL that compiles, Webpack building, but I couldn't get javascript include to work as it was building a separate manifest which means the javascript_pack_tag helper couldn't find it.

@Judahmeek

This comment has been minimized.

Copy link

Judahmeek commented Sep 18, 2017

Had any further luck figuring this out @mikecx?

@mikecx

This comment has been minimized.

Copy link

mikecx commented Sep 19, 2017

@Judahmeek Sadly not too much. Haven't spent too much more time trying to get it working but it would definitely be a nice to have for the bundle speed increases.

@p0wl

This comment has been minimized.

Copy link
Contributor

p0wl commented Sep 20, 2017

I got it working, the key is to run the vendor build (with DllPlugin) first and have that one write the manifest.json (using the webpack-manifest-plugin) first. The second build (webpacker with DllReferencePlugin) needs then to pick up the content of the already existing manifest.json using the seed option.

I will post our configuration tomorrow. We use it in dev, test and production.

@p0wl

This comment has been minimized.

Copy link
Contributor

p0wl commented Sep 21, 2017

Here is the configuration that we use.

config/webpack/vendor.js:

// Note: You must restart bin/webpack-watcher for changes to take effect
/* eslint global-require: 0 */

const { join, resolve } = require('path')
const webpack = require('webpack')
const ManifestPlugin = require('webpack-manifest-plugin')

const environment = require('./environment')
const { paths } = require('./configuration.js')

environment.plugins.set(
  'DLL',
  new webpack.DllPlugin({
    path: join(resolve(paths.output, paths.entry), 'vendor-dll-manifest.json'),
    name: '[name]',
    context: resolve(paths.output, paths.entry)
  })
)

const config = environment.toWebpackConfig()

// list of packages that will be served from vendor.js
config.entry = {
  vendor: [
    'classnames',
    'core-js',
    'lodash/flatten',
    // ... all packages that you want to remove from the entry bundle and ship it via vendor.js
    'redux',
    'redux-thunk',
  ]
}

// set library name to 'vendor'
config.output.library = '[name]'
config.stats = 'errors-only'

module.exports = config

config/webpack/environment.js

const { basename, join, resolve } = require('path')
const { existsSync } = require('fs')

const { environment } = require('@rails/webpacker')
const webpack = require('webpack')
const ManifestPlugin = require('webpack-manifest-plugin')

const { paths, publicPath } = require('./configuration.js')

const manifestName = join(resolve(paths.output, paths.entry), 'manifest.json')
const manifest = existsSync(manifestName) ? require(manifestName) : {}

const vendorManifestName = join(resolve(paths.output, paths.entry), 'vendor-dll-manifest.json')
const vendorManifest = existsSync(vendorManifestName) ? require(vendorManifestName) : {}

environment.plugins.set(
  'Manifest',
  new ManifestPlugin({ fileName: 'manifest.json', publicPath, writeToFileEmit: true, cache: manifest })
);

environment.plugins.set(
  'DLL',
  new webpack.DllReferencePlugin({
    context: resolve(paths.output, paths.entry),
    manifest: vendorManifest
  })
);

module.exports = environment

bin/webpack / bin/webpack-dev-server

add:

"#{NODE_MODULES_PATH}/.bin/webpack", "--progress", "--color", "--config", "config/webpack/vendor.js",

app/views/index.html.haml

be sure to load the vendor bundle first

    = javascript_pack_tag 'vendor'
    = javascript_pack_tag 'web_client'

running bin/webpack will result in a public/packs/manifest.json (which is picked up from webpacker) like

{
  "vendor.js": "/packs/vendor-dc001cc5d1289798a659.js",
  "web_client.js": "/packs/web_client-080286df8829595eea88.js",
  "vendor.js.map": "/packs/vendor-68f231280e5d468a447b.js.map",
  "web_client.js.map": "/packs/web_client-8d5d1fdc18ffced5e6ba.js.map"
}
@gauravtiwari

This comment has been minimized.

Copy link
Collaborator

gauravtiwari commented Sep 24, 2017

@p0wl Thanks for sharing. If you have time, perhaps you could open up a PR to document the usage
please 👍

@eriknygren

This comment has been minimized.

Copy link

eriknygren commented Nov 16, 2017

@p0wl thanks for sharing this, could you please elaborate on what configuration.js should include? I'm trying to get this working but I'm having some issues getting the paths right. Cheers.

@p0wl

This comment has been minimized.

Copy link
Contributor

p0wl commented Nov 18, 2017

configuration.js is pretty standard, no magic there:

// Common configuration for webpacker loaded from config/webpack/paths.yml

const { join, resolve } = require('path')
const { env } = require('process')
const { safeLoad } = require('js-yaml')
const { readFileSync } = require('fs')

const configPath = resolve('config', 'webpack')
const loadersDir = join(__dirname, 'loaders')
const paths = safeLoad(readFileSync(join(configPath, 'paths.yml'), 'utf8'))
const publicPath = `/${paths.entry}/`

module.exports = {
  env,
  paths,
  loadersDir,
  publicPath
}

and paths.yml as well:

# Restart webpack-watcher or webpack-dev-server if you make changes here
config: config/webpack
entry: packs
output: public
node_modules: node_modules
source: app/javascript
extensions:
  - .js
  - .jsx

What is not working for you? Maybe I can help when you post the errors that you get =)

@eriknygren

This comment has been minimized.

Copy link

eriknygren commented Nov 22, 2017

Thanks for sharing that @p0wl, I got my DLL pack compiling now. My problem now is that I'm not having any success executing the DLL build and then the dev server.

First I had

# bin/webpack-dev-server
 cmd = [
  "#{NODE_MODULES_PATH}/.bin/webpack", "--progress", "--color", "--config", "config/webpack/vendor.js",
  "#{NODE_MODULES_PATH}/.bin/webpack-dev-server", "--progress", "--color",
  "--config", WEBPACK_CONFIG,
  "--host", LISTEN_HOST_ADDR,
  "--public", "#{HOSTNAME}:#{PORT}",
  "--port", PORT.to_s
] + ARGV

Which it didn't seem to like at all.

Then I tried:

# bin/webpack-dev-server
dll_cmd = [
  "#{NODE_MODULES_PATH}/.bin/webpack", "--progress", "--color", "--config", "config/webpack/vendor.js",
] + ARGV

cmd = [
  "#{NODE_MODULES_PATH}/.bin/webpack-dev-server", "--progress", "--color",
  "--config", WEBPACK_CONFIG,
  "--host", LISTEN_HOST_ADDR,
  "--public", "#{HOSTNAME}:#{PORT}",
  "--port", PORT.to_s
] + ARGV

Dir.chdir(APP_PATH) do
  p "execing the dll"
  exec env, *dll_cmd
  p "execing the dev server"
  exec env, *cmd
end

I was thinking they need to be in separate commands, and this got me halfway there, but it never reaches the "execing the dev server"part now, looks like it just exits as soon as it runs the first command. So would be great if you could elaborate a bit more on your bin/webpack* files setup for us uninitiated 🙏

@TheRusskiy

This comment has been minimized.

Copy link

TheRusskiy commented Jan 24, 2018

@eriknygren did you get it working?

@TheRusskiy

This comment has been minimized.

Copy link

TheRusskiy commented Jan 24, 2018

@p0wl what version of webpacker is it for?
also, for me <script src="/packs/vendor-b6ff2e635b8d0a001b39.js"></script> gives 404 (it does exist), I figure that's because it's on disk, and not in memory? How were you able to get around that?

@TheRusskiy

This comment has been minimized.

Copy link

TheRusskiy commented Jan 25, 2018

Made it work with autodll-webpack-plugin.
Set up turned out to be more minimalistic, though a bit hack-ish and only got it working for dev so far (probably wouldn't want it on production anyway).
my webpack/development.js:

const environment = require("./environment")
const { join, resolve } = require('path')
const AutoDllPlugin = require('autodll-webpack-plugin');
environment.plugins.append("AutoDLL",
  new AutoDllPlugin({
    filename: '[name].dll.js',
    path: join(resolve("public", "packs"), 'vendor-dll'),
    debug: true,
    context: resolve("public", "packs"),
    entry: {
      vendor: [
        "actioncable",
        "...other packages to include"
      ]
    }
  })
)
module.exports = environment.toWebpackConfig()

then I had to also add to my layout this:

<% if Rails.env.development? %>
  <script type="text/javascript" charset="utf-8" src="/packs/vendor-dll/vendor.dll.js"></script>
<% end %>
@p0wl

This comment has been minimized.

Copy link
Contributor

p0wl commented Jan 25, 2018

We are still at webpacker 3.0.2, didn't find the time to upgrade yet.

Actually, I think it should be used in production because that way your users won't have to re-download your vendor bundle after each deployment, because the extract vendor libraries won't change.

In a setup without the dll-plugin the chunkhash of your generated assets changes when you change anything in your frontend code. So the browser will download all changed assets again.
With the dll plugin you separate libraries from application code. The assets that contain the libraries will not change when you change application code and as it is extracted it will remain with the same chunkhash so the browser does not have to download the libraries again.

without dll plugin:
packs/app.js contains everything, application code, react, ... and will be quite huge

with dll plugin:
packs/app.<chunkhash>.js will only include application code and therefore very small
packs/vendor.<chunkhash>.js will include all common libraries (react, ...) and be huge, but will only change when you change the version of a library or add one.

if your vendor.dll.js is missing from packs/manifest.json you have to find a way that it stays in there.
For us the critical point was to set the cache parameter for the manifest-plugin:

const manifest = existsSync(manifestName) ? require(manifestName) : {}
new ManifestPlugin({ fileName: 'manifest.json', publicPath, writeToFileEmit: true, cache: manifest })

but it seems that the auto-dll-plugin is not able to work with the manifest plugin correctly: asfktz/autodll-webpack-plugin#58

So if you only want to use it for dev it is fine, but if you want to improve long term caching of your assets in production, you should try to get it working without the autodll-webpack-plugin.

@TheRusskiy

This comment has been minimized.

Copy link

TheRusskiy commented Jan 25, 2018

@p0wl where I can I selectively import functions from packages, instead of importing the whole package. I may be wrong here, but to me it looked like DLL plugin just throws the whole package, so users are going to end up downloading stuff they don't need.
Also (and it's purely our use case) we may have several dashboards and each has it's own "pack" and their own libraries. Seems like a pain trying to configure dll plugin to make multiple caches for each dash, if we don't do it people are going to end up downloading packages from the whole other dashboard.

@eriknygren

This comment has been minimized.

Copy link

eriknygren commented Jan 26, 2018

@TheRusskiy I didn't. I couldn't figure out how to get it executing both commands, it kept exiting after running the dll build. So I reverted into relying on CommonChunks.

@p0wl any chance you could elaborate a bit on how your command executions look in your bin/webpack-dev-server file?

@faceyspacey

This comment has been minimized.

Copy link

faceyspacey commented Apr 5, 2018

here's a simplified version of code used to serve a hashed version of the dll:

class ApplicationController < ActionController::Base
  def auto_dll_src
    dir = Rails.root.join('public', 'packs')
    file = Dir.children(dir).detect { |f| f.index('autodll') == 0 }
    '/dist/' + file # serve from a different public path than webpack-dev-server
  end
  
  helper_method :auto_dll_src

  def send_gzipped_file
    response.headers['Content-Encoding'] = 'gzip'
    response.headers['Cache-Control'] = 'max-age=31536000'

    path = 'packs/' + params[:file] + '.gz' # get it from where webpack would normally put it (built prior to webpack-dev-server or with write-file-plugin)

    send_file(Rails.root.join('public', path),
      :content_type =>  'application/javascript',
      :disposition => 'inline',
      :status => 200
    )
  end
end

routes.rb:

get "dist/:file" => "application#send_gzipped_file"

application.html.slim:

script type="text/javascript" src=auto_dll_src

It's not as automatic as the convention-based stuff that webpacker is doing, but it finds files in the /packs folder and their hashes without the need for the manifest plugin to work. It also lets you serve the gzipped version, which you can make using the plugins option to autodll.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment