Reduce dependencies according to semver

[jhnns]

I haven't tried it yet, but does webpack reduce dependencies according to the package version? So if a module with the same version number is included from two different directories, e.g. A/node_modules/underscore@1.x and B/node_modules/underscore@1.x, webpack should include it only once into the bundle.

It would be even better if webpack includes only one version as long as the major version of those two dependencies don't differ. So if myLib1 depends on underscore@1.1.3 and myLib2 depends on underscore@1.0.2, webpack should include only underscore@1.1.3 because - according to semver - it won't break myLib2. Maybe this should be opt-in as it is potential risky.

In any case webpack should log a warning if two versions of the same module are included.

[sokra]

You can try npm dedupe, according to the documentation it should merge modules with the same (similar) version number. (Not tested yet)

Merging modules with different versions is very risky.

A similar feature is in my mind for a long time: Finding modules with the same code, but maybe different depedencies and merging them. At runtime I would include only a reference to the shared code and a dependency mapping table. It is a bit complex to implement, but this wouldn't break anything.

A small note: even merging modules with exact the same version can break something. If you rely on being the only user of you local module and the module has some static state. (very rar and bad style)

[sokra]

Definitly not in 0.9

[jhnns]

It is risky - that's why it should be opt-in. But I don't think that bundling two different versions of e.g. underscore for the client is the right thing to do.

But you're right: This could also be the job of npm when deploying the application. You may close this one or keep it open as a reminder.

Btw: components keeps the dependency hierarchy flat. I think, both styles - deep (npm) and flat (component) - are appropriate for their targeted environment.

[jhnns]

This will now become relevant as we're trying to split alamid in different modules.

[sokra]

This will come...

[sokra]

Here is the planned deduplication code. See it as riddle...

(function(m/*odules*/){
    for(var i in m) {
        switch(typeof m[i]) {
        case "number":
            m[i] = m[m[i]];
            break;
        case "object":
            m[i] = (function(fn) {
                var args = [].slice.call(arguments, 1);
                return function(a,b,c) {
                    m[fn].apply(null, [a,b,c].concat(args));
                };
            }.apply(null, m[i]));
            break;
        }
    }
})

EDIT: use Array.prototype.concat

[jhnns]

Looks good! Slice, unshift, apply - that's the way we go! 👍

[michaelficarra]

Should be concat.

[sokra]

I thought Array.prototype.concat isn't supported by IE6, but I think I was wrong... So it could be optimized a bit 😄 thanks

[jhnns]

Could you tell us a bit about your plans? I think this is a hot topic that needs to be discussed as it's not trivial.

[sokra]

I'll only perform safe optimization. If enabled a hash over each module source code (before dependencies) will detect duplicates. Dependencies can and will be different.

An example without dependencies:

• node_modules/a/file.js -> module id 1
• node_mdoules/b/node_modules/a/file.js -> module id 2

With the source:

module.exports = { value: 123 };

would result in

{
 1: function(module, exports, require) {
  module.exports = { value: 123 };
 },
 2: 1
}

The above code fragment than transform this array.

---

With dependencies it is a bit complexer:

• node_modules/a/index.js -> module id 3
• node_mdoules/b/node_modules/a/index.js -> module id 4

With the source:

module.exports = require("./file");

Would result in

{
 3: [5, 1],
 4: [5, 2],
 5: function(module, exports, require, __webpack_dependency_xyz__) {
  module.exports = require(/*! ./file */__webpack_dependency_xyz__);
 }
}

---

assert(require("a") !== require("b/node_modules/a"))

[jhnns]

So the code is not duplicated but require("a") returns a different instance than require("b/node_modules/a")? Is semver taking into account?

[sokra]

yes. No.

I think semver is too high level for webpack... It's working on file level, while semver is package level... semver is the job of npm.

[jhnns]

So how do you decide that two modules are the same? String comparison?

[sokra]

yes, maybe optimized with a hash. Maybe I'll compare after minimizing...

[sokra]

0.10.0-beta20 has experimental support. --optimize-dedupe

There is some stuff missing. Webpack may exit with a "Template cannot be applied as TemplateArgument" exception.

[jhnns]

Nice .. 👍

[sokra]

Here is an example: https://github.com/webpack/webpack/tree/master/examples/dedupe

[sokra]

And another with code splitting: https://github.com/webpack/webpack/tree/master/examples/code-splitted-dedupe

[jhnns]

So, first using npm dedupe to reduce the dependencies according to semver and then webpack's dedupe to remove duplication on file-level (if there are some remaining) will probably have the best result.