Dynamic Requires

[nick-thompson]

Hi!

My entry module uses a dynamic require, which forces Webpack to try to build every file under the context directory into one gigantic bundle. Is there a way to restrict which files webpack tries to bundle in this case? For example, ignoring any files underneath various test directories? Is there also a way to force webpack to split these files out into several bundles?

I think, because the dynamic require is so ambiguous, Webpack determines that every file within the context directory should be bundled with the entry file so that they're all available should the dynamic require need them. That makes plenty of sense, but at some point I'd like require to send off an ajax request for the next bundle. How can I force that?

Thanks!

[sokra]

There is a plugin for this. It overwrites the default RegExp (extracted from your dynamic require), with a custom one.

So if you have a file file.js in the folder folder:

require("./anotherFolder/" + expr + ".js");

The default RegExp would be /^\.\/.*\.js$/ for a context in .../folder/anotherFolder.

Within the configuration you could overwrite this:

var ContextReplacementPlugin = require("webpack/lib/ContextReplacementPlugin");
module.exports = {
  plugins: [
    new ContextReplacementPlugin(
      /folder[\\\/]anotherFolder$/, // change all contexts matching this RegExp
      /^\.\/(styles|templates|xyz)\/[^\/]+\.js$/ // Exchange the RegExp with this new RegExp
                                  // which matches only files in styles, templates and xyz
    )
  ]
};

---

The same behavior could be archived with the require.context method:

require.context(
  "./anotherFolder", // context folder
  true, // include subdirectories
  /^\.\/(styles|templates|xyz)\/[^\/]+\.js$/ // RegExp
)("./" + expr + ".js")

This would even generate exactly the same code and bundle.

---

To force the require to send off an ajax request, you need to use a asynchronous require variant (ajax is async). There are multiple options: require.ensure, require AMD and the bundle-loader.

/* Put the context into a chunk */
module.exports = function loadAsync(expr, callback) {
  require.ensure([], function(require) {
    try {
      callback(null, require("./anotherFolder/" + expr + ".js"));
    } catch(err) { callback(err); }
  });
};
/* or */
module.exports = function loadAsync(expr, callback) {
  require(["./anotherFolder/" + expr + ".js"], function(result) {
    callback(null, result);
  });
};

/* Put the context into the current chunk and each file into a separate chunk */
module.exports = function loadAsync(expr, callback) {
  var bundledResult = require("bundle!./anotherFolder/" + expr + ".js"]);
  bundledResult(function(result) { callback(null, result); });
};

/* Put the context into a chunk and each file into a spearate chunk */
module.exports = function loadAsync(expr, callback) {
  require(["bundle!./anotherFolder/" + expr + ".js"], function(bundledResult) {
    bundledResult(function(result) { callback(null, result); });
  });
};

The first and the second one create one chunk. The 3rd one creates N chunks (N = files in context). The last one creates N+1 chunks. You may want to automatically merge that many chunks with one of these options: optimize.minChunkSize, optimize.maxChunks

[nick-thompson]

Thanks for the detailed response!

/* Put the context into the current chunk and each file into a separate chunk */
module.exports = function loadAsync(expr, callback) {
   var bundledResult = require("bundle!./anotherFolder/" + expr + ".js"]);
   bundledResult(function(result) { callback(null, result); });
};

I think this is closest to what I'm looking for. Unfortunately my dynamic require statement is even more ambiguous than the one in your example:

require.ensure([moduleName], function () {
  var module = require(moduleName);
  module(args);
});

Thus, I think you're right, I'm looking at N+1 chunks. I would certainly like to merge those chunks into fewer chunks of larger size, but it seems to me that webpack can't correctly guess which chunks make sense to group in this situation. Is it possible for me to specify exactly which files I want bundled together in this option?

[nick-thompson]

Also, for what it's worth, when I try to use require.ensure I get an error saying that webpack doesn't have a method ensure.

[sokra]

The require.ensure doesn't accept expressions in it's frist argument:

// Do it this way (with an empty array)
require.ensure([], function () {
  var module = require(moduleName);
  module(args);
});
// Or this way:
require([moduleName], function(module) {
  module(args);
});

---

Are you sure I really need such a dynamic require. Isn't there another way solving your problem with static requires? Dynamic requires are difficult to handle and most time they include more stuff than intended (or require additional configuration). Maybe you can use a loader to extract the possible values before they get to the dynamic require.

---

You can write a plugin which merges the chunks the way you like it. Here is an example https://github.com/webpack/webpack/blob/master/lib/optimize/MinChunkSizePlugin.js

[jhnns]

@sokra is right. There are few situations where a dynamic require is necessary. But if you want to control the chunks, you could create modules that group those other modules in a logical way.

[fresheneesz]

@sokra So here's my problem. I'm working on a project where we want the ability for users to develop javascript plugins. These plugins will be pretty arbitrary, and potentially as numerous as users. Maybe the plugins will be stored in a database, maybe in the filesystem, but regardless, the application code itself won't know about the location of the file at the bundle-time of the main application.

It would be really nice if there was a way to create a bundle entirely separate from the main bundle in how its created, that could be loaded by a page that already has a main bundle loaded. Is that what bundle-loader does?

[sokra]

No this isn't possible with a webpack.

For security reasons you shouldn't compile untrusted code with webpack neither run it on your page. I'm not a security expert but my idea would be to run the user code in an iframe (in a separate domain) and give it access to some kind of API via cross-window messaging.

[fresheneesz]

@sokra So I understand the security concerns, however these would be modules a user explicitly requests running. So the user trusts them, even if the server doesn't. And since our server is going to be built properly (not trusting client's requests), I wouldn't think this would be an actual security problem. The user can screw up their client as much as they please - it's their client after all.

Are you saying there's no way to load disconnected modules/bundles via webpack's loaders?

Basically the way i'm imagining this would be done is:
A. User uploads a set of files which are bundled on the server OR the user bundles those files themselves and uploads the bundle(s)
B. When the user requests the (dynamic) module, the main bundle for that module is loaded and its module.exports returned in some way like this:

require("dynamic!http://somehost/some/path/to/someFile.js", function(module) {
// use module
})

I assumed that was what bundle-loader does. What does bundle-loader actually do?

[sokra]

webpack loaders "load" files while compiling. If you want to load files on runtime you need something like script.js (see #150).

If you compile on the server keep in mind that you can run arbitrary code (even native code) with something like require("./code.node!./file").

The bundle-loader loads modules in a separate chunk.

[fresheneesz]

@sokra Well when I say webpack "loads" files, I'm talking about split-points, when webpack actually does make a new request to the server. Good to know about the code.node loader.

I could certainly use something like script.js. What I would like to do ideally, is:
A. use webpack to create these user-modules (I suppose the safest way is to have the users' bundle them). This would require that webpack bundles can expose the main module in some way so that the user can access it after being loaded by script.js.
B. not have to load multiple copies of the webpack runtime (to save space and avoid any issues created by having multiple instances of webpack running). This would require some option that would allow external scripts to access and use webpack globally, and thereby use the runtime to load the separate bundle's modules.

Is there a way to use script.js to load webpack bundles with one or both of those things ^ being the case ?

[fresheneesz]

I'm also not sure what you mean that bundle-loader loads modules in a separate chunk. Do you mean that when webpack builds the bundles, it will explicitly bundle that single file as a bundle on its own?

[jhnns]

The bundle-loader just creates a split-point. That's the place where webpack will do a new request to the server. Given you have two modules a.js and b.js:

• a.js

require("bundle!./b.js")(function(b) {
    // now you can use the b-module
});

• b.js

module.exports = 42;

a.js will be in the first chunk, b.js will be in second chunk. Without the bundle-loader, a.js and b.js would be in the first chunk. A chunk is just a file containing multiple modules.

[fresheneesz]

Hmm, so what's the difference between the bundle-loader and require.ensure? Just shorthand for

require.ensure(['something'],function() {
   var b = require('something')
})

?

[jhnns]

Yep. And there is the lazy option:

var fn = require("bundle!./a.js"); // require() triggers the request

var fn = require("bundle?lazy!./a.js");
fn(); // fn() triggers the request

[fresheneesz]

Ah gotcha. Thanks.

[fresheneesz]

@sokra Am I wrong that this shouldn't be very complicated to implement? Instead of integers, urls could be used as the keys for these dynamic modules. There would need to be a way to build a bundle intended to be loaded in this way (which omits the runtime that already exists in the entrypoint bundle) - basically this would be creating a standalone split-point bundle.

This would open up the possibilities for a ton of different mashups.

Using iframes and cross-iframe messaging is not an acceptable solution in this day and age. And the alternative (using a loader like $script.js) is also hacky and sub-optimal. It would need at least one global variable to be used to manage loading other files. And if you want to use webpack for each of these, it also runs into the possibility of conflicts because of multiple instances of the webpack runtime (e.g. the jsonp function name). I've already explained why security isn't an issue here. So why not aspire to support this?

Can we please reopen this issue?

[sokra]

We cannot reopen this issue as this is unrelated to the original issue.

It's not possible to add an additional chunk to a bundle. They need to know eachothers module ids. So they need to be build in one build step.

You can easily avoid the confilicts of multiple webpack bundles in one page by adding a UUID to the jsonp function name.

So your plugin developers compile a plugin bundle with this webpack.config.js:

var UUID = createUUID(); // i. e. '110ec58a-a0f2-4ac4-8393-c866d813b8d1'
var UUID_ident = "_" + UUID.replace(/-/g, "_");

var webpack = require("webpack");
module.exports = {
    entry: "./test",
    output: {
        path: __dirname + "/dist",
        filename: "main.js",
        chunkFilename: "[id].js",
        publicPath: "/plugins/" + UUID + "/",
        libraryTarget: "jsonp",
        library: UUID_ident,
        jsonpCallback: "webpackJsonp" + UUID_ident
    },
    plugins: [
        new webpack.BannerPlugin("xyz plugin [" + UUID + "]")
    ]
};

And upload the dist folder to your server /plugins/<UUID>. Now you can load plugins with the UUID like so:

function loadXyzPlugin(uuid, callback) {
    var uuidIdent = "_" + uuid.replace(/-/g, "_");
    window[uuidIdent] = function(exports) {
        delete window[uuidIdent];
        head.removeChild(script);
        callback(null, exports);
    });

    // start loading via jsonp
    var head = document.getElementsByTagName('head')[0];
    var script = document.createElement('script');
    script.type = 'text/javascript';
    script.charset = 'utf-8';
    script.src = "/plugins/" + uuid + "/main.js";
    head.appendChild(script);
}

Instead the UUID you could also use a unique name generated by or registered on your server. For mutliple versions you can append the version string.

The server can extract the UUID from the updated files. It's added in a comment through the BannerPlugin.

[fresheneesz]

@sokra

"They need to know eachothers module ids."

So if module A, requires module B, module A needs to know the module-id for module B, but does module B need to know module A's module-id? I wouldn't think so.

"The server can extract the UUID from the updated files."

I was actually thinking I could scrape the jsonp name using Esprima. Definitely more complicated, but requires less from users (who would submit these extensions).

[KidkArolis]

@fresheneesz hey, did you end up using webpack for your project? Did you run into any further issues? How did you solve the dynamic loading?
We're building something similar and I'm also exploring the different ways of dynamically loading in extra user modules into the page. At the moment we're using require.js which works quite well since you can extend the require.config at runtime and pull in the extra modules asynchronously. But I'm intrigued by the extra capabilities that webpack would provide (cjs, css, perf, npm).

[fresheneesz]

@KidkArolis We are using webpack for our project, and its been working very well. Its not solving the dynamic loading problem though. I started writing a small extension loading module using scriptjs. We haven't gotten the point of needing it yet so its still unfinished, but we do plan on finishing it. If you don't want to put in the work for something like that (its not too difficult) and don't care about the extra size of additionally using require.js, you could theoretically use require.js side-by-side with webpack - webpack loading files known statically and require loading files dynamically.

[kellyrmilligan]

Is there a way to set what the base url is for the ajax request for loading the bundles on the fly? I think I have my config as I would like it, but it's requesting the files at the root of my app, which is not where they end up after the build process.

[kellyrmilligan]

sorry, this is with the output config for public path, correct?

[guillaume86]

I'm also interested in the problem.
I'd like to implement a plugin architecture, plugins should be builds into separate bundles and loaded dynamicly (something like Atom Editor for example) and should reuse modules available in the core bundle.

The root issue seems to be that webpack build configs are not composable:
In browserify you can configure a bundle (equivalent to a webpack compiler instance) to expose modules X and Y and then you set it as an external for the second bundle and the second bundle will know it can load the core modules from the main bundle and not duplicate packages.

[sokra]

@guillaume86 You could do the same with webpack (http://webpack.github.io/docs/webpack-for-browserify-users.html#external-requires + externals: {"core": "commonjs core"}) but I wouldn't recommend this.

[fresheneesz]

Ooo, that's very interesting. Why wouldn't you recommend that?

[sokra]

It would be a pretty complex and browserify/webpack-specific API for something that could be solved with a simple global Core var too. A simple global var would allow to write plugins without a module bundler too. As simple scripts.

I'm not happy with a globally exported require function that looks like a CommonJs require, but isn't CommonJs-compatible.

[fresheneesz]

@surkova You can use require.ensure to enable dynamic loading in your case

[mschipperheyn]

Just curious, why a callback in stead of a Promise? Seems kind of old school

[SimenB]

Webpack 2 will suport System.import, which returns a promise. See #861

[adamziel]

Is there a way to make webpack behave a bit like require.js and load each and every file asynchronously in development? This would eliminate the need for source maps that are the root cause of many debugging troubles (some stack traces are not mapped for example)

[jhnns]

@adamziel I think that would be pretty complicated to do and thus it out of scope (and it drastically changes the way how your code works in development).

[juan-bastidas]

Hi

I have a folder with many files, those are libs to load different shapes on an application to draw things. Normally we don't need all of them loaded at once, they are needed on demand.

I'have defined a split point for all files within that folder. I load those files dynamically at runtime, but not all of them are needed just few depending on the case.
The folder contains 64 files, all of them 9MB and again at runtime I load 5, maximum 10.

I'm moving from requirejs world to webpack and with requirejs I see on the browser just few of them loaded, with webpack the problem is that one bundle is created, that bundle contains all files so the bundle is 9MB. I want to not have them all in just oen bundle, I've tried many things and few plugins like webpack-split-by-path and split-by-name-webpack-plugin but still one bundle is generated. How can I configure webpack to make more bundles or even generate one bundle per file just for that folder?

Any advice?

Thanks

[somebody32]

@juan-bastidas I recently wrote an [article about that](https://medium.com/@somebody32/how-to-split-your-apps-by-routes-with-webpack-36b7a8a6231#.dy38h6mqr/
There I'm splitting app by routes, but I think it can be easily converted to your case

[jbielick]

@juan-bastidas I have a similar setup and this was indeed possible with @somebody32 's help.

For example:

We load templates asynchronously and we only need one or two. There are dozens, though. One template can be loaded like this and will be bundled completely separately. If you're also using the CommonChunksPlugin, these isolated pieces will only contain their specific content and not the common code.

Install bundle-loader and then:

    let getTemplate = require(`bundle!./templates/${templateName}.jsx`);

    getTemplate((module) => {
      let preview = React.createElement(module.default, data);
      ReactDOM.render(preview, this.el);
    });

[mcku]

I have a similar situation as @somebody32 mentioned where app is split by routes and dynamic requires. The interesting thing is, output bundle contains unminified strings wrapped by eval(). Is there a way to make sure it gets minified, or am I missing something?

example:

/* this is okay to me */
!function(e){function n(t){if(r[t])return r[t].exports;var a=r[t]={exports:{},id:t,loaded:!1};return e[t].call(a.exports,a,a.exports,n),a.loaded=!0,a.exports}var t=window.webpackJsonp;window.webpackJsonp=function(r,o){for(var i,s,l=0,c=[];l<r.length;l++)s=r[l],a[s]&&c.push.apply(c,a[s]),a[s]=0;for(i in o)e[i]=o[i];for(t&&t(r,o);c.length;)c.shift().call(null,n)};var r={},a={1:0};return n.e=function(e,t){if(0===a[e])return t.call(null,n);if(void 0!==a[e])a[e].push(t);else{a[e]=[t];var r=document.getElementsByTagName("head")[0],o=document.createElement("script");o.type="text/javascript",o.charset="utf-8",o.async=!0,o.src=n.p+""+({0:"chp",2:"fp",3:"csp",4:"lp",5:"clp"}[e]||e)+".chunk.js",r.appendChild(o)}},n.m=e,n.c=r,n.p="",n(0)}

/* why eval ?  */
([function(module,exports,__webpack_require__)
    {eval("\"use strict\";\nvar __extends = (this && this.__extends) || function (d, b) {\n    for (var p in b) if (b.hasOwnProperty(p)) d[p] = b[p];\n    function __() { this.constructor = d; }\n    d.prototype = b === null ? Object.create(b) : (__.prototype = b.prototype, new __());\n};\nvar __assign = (this && this.__assign) || Object.assign || function(t) {\n    for (var s, i = 1, n = arguments.length; i < n; i++) {\n        s = arguments[i];\n        for (var p in s) if (Object.prototype.hasOwnProperty.call(s, p))\n            t[p] = s[p];\n    }\n    return t;\n};\nvar React = __webpack_require__(1);

... the whole module continues without minification ...

Is this above how dynamic require is intended to work, how can i push more uglifyjs2 into it?

[mcku]

Well, I could not find a working means in webpack but the workaround was to check environment conditionally in the code such as:

    require.ensure(['./myModule'], (require) => {
        resolve(require(
          (process.env.NODE_ENV === "production"?'uglify!':'')+
          './myModule'
        ).default);

this one results in the same eval wrapped strings, but this time they are uglified

[sokra]

It's wrapped by eval because you used an eval devtool.

[kentmw]

@fresheneesz did you end up solving your problem of wanting to require user defined plugins using webpack? I am starting to design a plugin architecture and I'd like the person building the application (using webpack) to define an plugin config, something like:

plugins = [
  {
    package: "myPlugin",  // or "../path/to/plugin/myPlugin"
    // other config
  },
  {
    // ...another plugin
  }
]

and in the main application, I'm hoping to do:

plugins.forEach(function(plugin) {
  require(plugin.package);
});

@sokra Because the person defining the plugin config is the same person running webpack to build the end bundle, I don't see why this would be a security issue.

[fresheneesz]

@kentmw I haven't yet, but I will soon (in the next month or so). In my case, the person defining the plugin config will not be the same person building the end bundle, so I just won't be using any webpack functionality directly for this.

[jri]

Thanks to sokra's Apr 17, 2014 comment the problem of loading independently compiled bundles is now solved for me. I describe a slightly adapted solution for Webpack 2 here:
https://stackoverflow.com/questions/43163909/solution-load-independently-compiled-webpack-2-bundles-dynamically

[luisherranz]

That's really interesting @jri, thanks for sharing.

I ended up doing the same, but using comonjs for library output and systemjs to import the plugins. I wouldn't recommend Systemjs, it has given me some headaches so I guess your JSONP method is better.

One of the problems I found was the access to the common chunks, so libraries like react are not included on each plugin. I ended up using Webpack's DLL. The problem here is that each time you update or change this vendors bundle the chunk ids get messed up and you need to rebuild all your plugins arsenal. I use FixModuleIdAndChunkIdPlugin to solve this but it has giving me some troubles when combined with DedupePlugin. You also need to retrieve this vendors bundle and keep it in sync with everything.

After a while using this system I wonder if it's worth the effort or if it would be better to switch back to "one webpack" configuration and let the backend server retrieve all the plugins and build everything at once each time.

This is the code just in case anyone wants to take a look:
https://github.com/worona/worona-cdn/tree/master/packages

[lili21]

@sokra

webpack loaders "load" files while compiling.

if (!window.Promise) {
  window.Promise = require('promise-polyfill')
}

how this one work ?

[jedwards1211]

@fresheneesz

I'm working on a project where we want the ability for users to develop javascript plugins. These plugins will be pretty arbitrary, and potentially as numerous as users. Maybe the plugins will be stored in a database, maybe in the filesystem, but regardless, the application code itself won't know about the location of the file at the bundle-time of the main application.

I think now you could use DllPlugin/DllReferencePlugin to accomplish what you're trying to do (if I understand correctly, you want a way for a plugin bundle to require modules out of your main application bundle). This same kind of thing has been on the back of my mind for some time...

@luisherranz did comment on the diffculties of this approach.

[fresheneesz]

Thanks for the note @jedwards1211, I'm a bit out of that world at the moment, but I'll have to check it out when I get back in!