Unexpected behavior with linked NPM modules

[ooflorent]

tl;dr

Webpack should produce the same build whether a module is linked or not.

In depth

Considering the following dependency trees:

project@1.0.0
├── babel-runtime@5.1.9
└── sub-project@1.0.0

sub-project@1.0.0
└── babel-runtime@5.1.9

Notice: NPM does not include babel-runtime under sub-project when npm list is called from project.

Webpack builds something link:

[0] multi main 52 bytes {0} [built]
[1] ./index.js 2.08 kB {0} [built]
[2] ./~/babel-runtime/helpers/interop-require-wildcard.js 148 bytes {0} [built]
[3] ./~/sub-project/index.js 2.46 kB {0} [built]

If we run npm link sub-project we obtain the following tree:

project@1.0.0
├── babel-runtime@5.1.9
└── sub-project@1.0.0 -> /Users/fc/Development/sub-project

Notice: babel-runtime still doesn't appear under sub-project.

But building the project again will duplicate its dependencies:

[0] multi main 52 bytes {0} [built]
[1] ./index.js 2.08 kB {0} [built]
[2] ./~/babel-runtime/helpers/interop-require-wildcard.js 148 bytes {0} [built]
[3] /Users/fc/Development/sub-project/index.js 2.46 kB {0} [built]
[4] /Users/fc/Development/sub-project/~/babel-runtime/helpers/interop-require-wildcard.js 148 bytes {0} [built]

Using resolve.root or changing the resolve scope of loaders do not change anything.

Expectations

It would link to be able the link my in-development modules into my project using npm link without having to tweak the webpack config.

The real problem isn't the build size but the unexpected behaviors raised but duplicated dependencies.

[ooflorent]

See #943 for related discussions. cc @sokra @magsout @Nyalab

[jhnns]

Mhmmm ... we've discussed about that already but I can't find the issue anymore. The problem is that linked modules have another resolved filename and I'm not sure if we can just change that. For instance, node's __filename also resolves to the real path...

[ooflorent]

Problem is React uses some global definitions and static objects (such as ReactInstanceMap). When included twice, React is unable to figure out if the component exists. In some complex cases using context, it totally fails.

[sokra]

I see the problem, but I don't know how to solve it. webpack implements the node.js resolving algorithm: https://nodejs.org/api/modules.html#modules_all_together
This describes how modules are handled and which places should be looked up for dependencies (that's not the application node_modules folder).

So the problem is by design and should occur in node.js too.

A workaround is to let webpack prefer the application node_modules with: resolve.root and resolveLoader.root = path.join(__dirname, "node_modules")

[ooflorent]

Unfortunately settings resolve.root and resolveLoader.root does not solve the issue.

Hash: 6d255d7e6eb942c864e9
Version: webpack 1.8.4
Time: 3896ms
                    Asset       Size  Chunks             Chunk Names
               index.html  345 bytes          [emitted]
                  main.js    4.71 MB       0  [emitted]  main
chunk    {0} main.js (main) 1.67 MB [rendered]
    [0] multi main 52 bytes {0} [built]
...
[478] ./~/react/lib/toArray.js 2.08 kB {0} [built]
...
[522] /Users/fc/Development/xxxxxx/~/react/lib/toArray.js 2.08 kB {0} [built]
webpack: bundle is now VALID.

[johnelliott]

I am having this issue too. At some point my changes to my module stopped rebuilding the webpack bit of a project I was integrating it with.

[jasonslyvia]

Hate to bother but is there any update about this issue?

[MoOx]

the workaround explained using npm link to get one version of react only (on both webpack and node) helped me #966 (comment)

[johnelliott]

@jasonslyvia I got around this too that same day. I had to gut the linking and dump every cache and node module I could find. I still don't know if the root of the issue was this code or my fault.

[jessepollak]

I have this same issue — the fix that @MoOx specifies works, but I have packages with a bunch of shared dependencies, so it's not a viable long term solution.

[chrisbateman]

Preferring the application node_modules with resolveLoader.root worked well enough with npm 2 - but with npm 3 it means that you will randomly pull in the wrong version of a package when there are duplicate versions within the app.

For instance - say my app uses moduleA which depends on lodash@3 and moduleB which depends on lodash@4. The npm3 install ends up looking like this:

• lodash@3
• moduleA
• moduleB
  • lodash@4

If I set resolveLoader.root - moduleB ends up pulling in lodash 3 rather than 4 (since lodash@3 just happened to get installed at root).

[jure]

So for this particular problem, there are now two useful solutions (but quite tricky ones):

1.) npm linking the linked module's dependencies to the project's dependency, i.e. #966 (comment)

2.) sharing a node_modules folder between linked module and project, i.e.
#554 (comment)

I don't like either of those due to their manual/repetitive/unstable nature. I'd prefer this solution: aurelia/webpack-plugin#44 (comment), a webpack ResolverPlugin that replaces the linked modules node_modules requires with the project's. That's what we're attempting. Any other solutions out there?

Update: (thanks @ganmor #985 (comment))

3.) not using npm link at all for development, but instead using some kind of file syncing tool, i.e. https://github.com/wix/wml or http://www.freefilesync.org/

[ganmor]

One possible solution is not using npm link at all.
If you are not using windows, here is a nice alternative https://github.com/wix/wml

[lazd]

Uhg this issue is plaguing me right now. The second I link a package in that has a dependency in common with the parent project, my build contains duplicates.

[niieani]

I've written a resolve plugin which takes care of this issue.

What is does is tries to resolve any module calls to the closest one to the provided rootDir, always preferring local ones, as long as the version of the closer dependency is within the requested package's SemVer range. Useful especially when using linked modules, but even when not using linked modules it it's pretty useful, as you don't end up with duplicate copies of the same package if another copy is within a nested node_modules.

It's called RootMostResolvePlugin and it's a part of the webpack-dependency-suite (sorry, no docs yet).

Configuration is pretty simple (example).

Make sure to put the plugin in the resolve.plugins array of your configuration (not plugins).

When instantiating, the first parameter should be the root directory of your project (or the one in which context you want to resolve) and the second, optional parameter is whether you want to enforce it for all paths (e.g. including linked modules, use true), or only for nested node_modules (false).

[jure]

Hey, just a quick update on our side. We've managed to solve our problems using npm linked dependencies by:

Using Webpack 2 and setting

  resolve: { 
      symlinks: false,
      modules: [
        path.resolve(__dirname, '..', 'node_modules'),
        'node_modules'
      ],
  }

So the local/root node_modules is always preferred, and symlinks are not resolved, i.e. same as the node --preserve-symlinks options, this was implemented in: #2937

This is working as expected for us, if that helps anyone stumbling upon this issue.

[niieani]

@jure, you're opening your project up to potential problems if you get many major/minor versions of the same package dependent on a different one, because resolve.modules does not verify the version of the package. If you want to mitigate that and ensure you are only including modules that match the requesting module's SemVer constraint, you can use the resolve plugin I mentioned before.

[bebraw]

Can someone check this against webpack 2? Thanks.

[niieani]

@bebraw The problem described in the issue is still present in Webpack 2, and as far as I understand will not be fixed, because the dependency resolution algorithm is based on the one provided by Node. Webpack works properly, but it's an arguably bad design decision on Node's part that Webpack follows. A webpack resolve plugin can be used to mitigate the problem successfully.

[travi]

just to provide a bit more insight of what is working, i was able to get this working in webpack 2 with a similar resolve config as above. thanks @jure!

resolve: { 
  symlinks: false,
  modules: [path.resolve('node_modules')],
}

it seems like it would pretty safe to default to the local node_modules/ without needing to configure resolve.modules, maybe based on context?

[trusktr]

Although it would stray from the standard Node.js resolution algorithm, I strongly believe Webpack should be smarter here (at least behind an option) and prefer packages found in the app's node_modules (the root-most node_modules) if those packages meet the semver requirements of the other packages depending on them.

@niieani's RootMostResolvePlugin (above) does exactly this.

@sokra Do you think it would be worth making this behavior the default in Webpack?

[trusktr]

@niieani I tried RootMostResolvePlugin with Webpack 4.16.4, and I get

Error: Plugin could not be registered at 'resolved'. Hook was not found.

Does it need an update? Does it need to use the afterResolvers hook?

EDIT: Oh! It needs to be placed in webpackConfig.resolve.plugins, not webpackConfig.plugins.

[goatfryed]

Hey guys,
I ran into this issue while trying to develop a library feature with an optional dependency. I noticed that my demo project - although not requiring the dependecy - still got it bundled. So i removed the optional dependency from my libs module folder, which broke the second demo project that should have had this dependency.
It took my quite sometime and testing until i finally found this issue and #811.

Currently, this is mentioned in https://webpack.js.org/configuration/resolve/#resolve
What do you guys think about adding a passage in https://webpack.js.org/guides/author-libraries/ as well?

[vankop]

Should work with webpack@5. Feel free to report new issue with reproducible repo.

[fabis94]

@vankop is there a way to get this functionality in webpack@4 as well?

[vankop]

no, only critical fixes for webpack@4 is possible..

[fabis94]

@vankop are there any docs about how this is supposed to work in webpack@5 or what are the limitations? cause I'm still running into the same issue - three.js is loaded both from the package's own node_modules dir and also from a symlinked local packages viewer node_modules dir (so ./node_modules/three and ../viewer/node_modules/three)

[vankop]

@fabis94 from your description is unclear what is wrong..

three.js is loaded both from the package's own node_modules dir and also from a symlinked local packages

maybe some dependency ( or different app modules ) is trying to resolve three.js and resolve wrong one..

Probably in your case problem is not related to symlinks.. ( it will work same way with non-symlink version ) you can reduce this to reproducible repo I will take a look.

[vankop]

I read the description once more.. I thought this related to symlinks..

Issue still exists

---

Basically #985 (comment) is an answer. Since webpack implements Node.js resolve algorithm there is no way to solve this..

[fabis94]

Just an idea, but why not just compare if the same version of the same package is already bundled in? Or make that an opt-in feature? Like for example, check first three.js package.json to resolve version, and then when three.js is about to be bundled in from another location, check the package.json again and stop doing so if it's the same.

The underlying issue is that after linking in modules, npm doesn't deduplicate the node_modules tree so you can end up having /node_modules/three and ../symlinked-package/node_modules/three both available. And then when three is imported from symlinked-package it will use its own local three version, but when three is imported by the host application built by webpack it'll use /node_modules/three, resulting in 2 three.js versions bundled in at the same time.

But this seemingly npm-specific issue is definitely not going away, considering that symlinking packages into node_modules is a very common workflow. So I think it makes sense for the improvement to be introduced on the webpack side, instead.

[vankop]

"fix" is possible, but this breaks Node.js resolution algorithm.

[fabis94]

I don't really see why people would want the current behaviour (same package with same version being bundled twice), but it could be an opt-in feature, if you think it's gonna cause issues for some.

Anyone trying to deal with these issues and spending hours on it not getting anywhere isn't going to care that the Node.js resolution algorithm is broken, getting dependencies to be properly bundled without duplicates is more important. And of course - by opting in you'd understand what you're doing and the consequences.