symbolic link duplicate module load

[gorangajic]

I am using handlebars-loader and when node_modules folder is linked from some other location Handlebars.SafeString is loaded from two location moduleA.SafeString and moduleB.SafeString

bundle if you search inside of this bundle you will find two instance of function SafeString

I have created demo so you can reproduce
https://github.com/feroc1ty/webpack-symbolic-link-bug

[jhnns]

@sokra webpack should probably resolve symbolic links with fs.realpath?

[gnarf]

We have also just spent a good while debugging a problem related to this. We have some shared code that is symlinked into a couple different repos, we use an internal_modules/module-name folder that is a symlink to ../module-name -- So when combining everything together, if two modules require('module-name') it actually gets loaded twice, and there are two distinct objects. I would expect that it would de-dupe the dependency correctly across symlinks

[MSeal]

This issue destroyed my productivity for a whole day once the race condition for loading modules caused two Handlebars to load in a particularly bad order. It isn't obvious at all that syn linking would cause anything like this issue so it's very painful to debug. Please push a fix so you don't cause more strangle-worthy frustration for other developers.

[jhnns]

@MSeal calm down. The solution is not as clear as you think. Maybe someones is relying on that webpack handles them as distinct paths...

[MSeal]

Well a warning or notice somewhere would at least be a good first step if there's duplicate paths being generated because of symlinking. Regardless of if someone is relying on duplicate pathing, it's hard to argue that double loading handlbars is the correct behavior as you wipe out the global state after the race condition resolves.

[irvinebroque]

Ran into this as well -- particularly tricky for isomorphic apps because npm link is one of the recommended "node ways" of dealing with local dependencies. For example, in node, if name: "myApp" in your package.json:

npm link && npm link myApp

You can require a module like this:

var myModule = require('myApp/dir/dir/myModule');

This is really nice for large apps, because you don't have to deal with relative paths. But as noted above, it creates a ton of duplicate paths in Webpack (over 1000 in my case), which doesn't behave the same as NPM here.

[jhnns]

@irvinebroque I see... but if you used the non-relative style throughout the whole app, there wouldn't be duplicate paths.

@sokra what do you think? Should they be resolved? I think, they should...

[sokra]

I think this can be solved...
But maybe there are some edgecases we need to consider...
Or propably it's enough to call realpath after resolving...

[irvinebroque]

@jhnns - true, but that means if one person uses a relative path by mistake, it causes duplicate requires without any warning or error. Basically setting yourself up for something failing silently.

Even though I don't agree with the way NPM handles this with symlinks, IMHO it's really important to be able to mimic NPM's behavior. I may be wrong here, but I think most people are more comfortable adding extra configuration to webpack.config than modifying the way require() works in node (via enhanced-require or compiling server code with webpack --target node.

[jhnns]

@irvinebroque I agree.

[maoziliang]

The symbol link problem is still exists. I already update webpack to 1.10.1.
My situation:
project A depent on react. npm link material-ui in project A. Also material-ui dependent on react. After webpack compile, duplicate react modules exists in the bundle file.

[arypurnomoz]

i'm using 1.10.5 and also encounter this issue

[maoziliang]

Could anyone solve this problem. Now I install local package instead of npm link to debug. It's very inconvenient.

[jedwards1211]

I'm confused how this works...

I'm trying to develop a bugfix on a fork of google-map-react. I've used npm link to link it into my Webpack project that actually creates a site with an instance of a <GoogleMap/>. google-map-react is built with Webpack with react declared as an external, and both it and my other project have React 0.13.3, but still it appears that google-map-react is loading a second copy of React from its own node_modules folder. Does anyone know how to work around this?

[Zenwolf]

I have the same problem where a second copy of react is bundled along with the symlinked module (via npm link) even though the parent module already has it installed. If I delete the react module from the symlinked module's node_modules dir, webpack errors out because the symlinked module can no longer find react and apparently webpack doesn't detect that the parent already has it installed. Both have the same version of react, 0.13.3. I am using the latest Webpack 1.12.0.

[arypurnomoz]

npm dedupe solved this in my case

[timmfin]

FYI, for others that come here. I was in a situation like @jhnns mentioned, where I'm relying on not resolving the symlinks (otherwise relative paths inside a symlinked module didn't work, because the resolved symlink ends up in a different directory than the file referred to by the relative path).

Fortunately, I figured out how to get around this by writing a Webpack plugin that prevents ResultSymlinkPlugin from running: timmfin/broccoli-webpack-cached@5dfcd31

[Agamennon]

I have been in the same predicament from others in here, i am npm linking a project that has react as external and its duplicating it, how are you guys solving this?

[SonofNun15]

Ran into this issue as well. I'm using npm link on a child library. Both the child library and the parent application reference the same external library which is being bundled into the output multiple duplicate times.

[vvo]

Ran into this issue as well. I'm using npm link on a child library. Both the child library and the parent application reference the same external library which is being bundled into the output multiple duplicate times.

Exact same problem right now.

[sebastiandedeyne]

Are there any workarounds for this yet?

Trying to set up a dev environment for one of my packages, which is required by a larger project, which in turn contains a gulp build process with webpack.

Using npm link is resolving some dependencies twice.

[Zenwolf]

One workaround I've found is that you have to go into your parent project's node_modules and npm link all the dependencies that are shared with the modules you want to work on locally.

Example using React library:

1. cd parentProj/node_modules/react.
2. npm link.
3. cd myLocalModuleThatParentProjUses/node_modules.
4. npm link react.

Then you are guaranteed that both the parent project and the child module you are working on locally will use the same lib. After you are done make sure to unlink everything. It's a huge headache when developing locally with modules. As much as I love the flexibility of Webpack, I never had these symlink/duplicate issues when using Browserify. I hope that this information helps people with this problem.

[SonofNun15]

We are working around this by npm installing the local git repo instead of using npm link. Our repositories are siblings in the main folder, so instead of using npm link, we make changes in one project and then in the other project: npm install ../other-project. This installs the local folder including all changes instead of installing from the repository in npm. Once you solidify the changes you can publish the child and run npm install other-project --save to make the changes in the parent official.

The only drawback we have seen is that we sometimes forget to update the parent's package file / dependencies before merging changes to the parent.

[vvo]

cc @neoziro is your shiny new module fixing this? if so, how?

[gregberge]

@vvo my module only watch a folder and reinstall it in your current project if a file change, it's simple and it fixes all npm link issues. It's exactly the solution proposed by @SonofNun15.

So if you want to automatize it : https://github.com/neoziro/npm-sync

It's currently not very robust and stable. But I am open to every contributions!

[andrewwebber]

Within the context of npm link and react development the following worked for me as a work around:

resolve: {
    modulesDirectories: ["node_modules"],
    alias: {
      "react": NODE_MODULES_DIR + "/react",
...

PS: I only have this issue w$ndows

[jessepollak]

I'm having this same issue with react in a npm link'd dependency. Has anyone figured out a solution that reliably works?

[SonofNun15]

Well, our current solution is to use a file sync tool (Free File Sync: http://www.freefilesync.org/) to copy the source folders into the npm dependency. This was a fast and easy way to get the desired functionality without the headaches of npm link and / or local npm install.

[bitjson]

@jessepollak (and for anyone else struggling with an npm linked dependency)

Until this is addressed in a more permanent way, I've found a really good solution is to share a single node_modules directory between the two dependant packages of the symbolically-linked dependency.

It sounds a bit hacky, but after you get over the shock, it comes with a number of benefits. As long as your packages share the same dependency version requirements (i.e. one doesn't rely on an outdated module), simply creating a symbolic link to the first package's node_modules directory before installing the second package eliminates the issue entirely. Likewise for as many packages as needed. (It's actually very possible to do this even with some clashing versions, though that requires some remapping of names.)

I've found this is particularly valuable for monorepo projects with multiple similar packages because it also slashes install time and total dependency weight significantly. See bitpay/icv2/package.json for our post-install script (and a working example monorepo).

Basically:

cd first && npm install && cd ../
cd second && ln -s ../first/node_modules/ node_modules && npm install && cd ../
cd third && ln -s ../first/node_modules/ node_modules && npm install && cd ../
[...]
cd first && npm link my-module

Each npm install adds it's dependencies to the symlinked node_modules directory. Now you have a single directory of dependencies shared across the entire project, with no chance of strange interactions caused by my-module being a symbolic link.

[viskin]

@timmfin proposed solution that worked for us very well. I want not resolving symlinks to be default behavior of webpack.

[flyon]

based on the suggestion of @timmfin I've come to a slightly different setup that's using for me. Instead of having a first module as the main source of a single shared node_module folder, I've made the parent directory of all modules a module with npm init and then npm link to all the modules once from there. Then each module that likes to include another module will not require anything to be done because node's module resolution will find it in the parent directory.

[Relequestual]

I looked at trying to implement @timmfin's solution, but I couldn't see how it would slot in or work with exsiting projects I'm working with. The simpler solution was to npm link the react in the parent project to the child project, as per #554 (comment) - Figure others may find this information useful.

[Venryx]

I also tried using @timmfin's solution, however it failed for my project, because my project uses Webpack 4.41.2, and Webpack 4 does not have a resolver.plugin("result", ...) hook.

You could probably achieve the same thing by swapping .plugin("result", ...) with .getHook("file").tapAsync(...) (as per the latest SymlinkPlugin source), but I figured I'd take a different approach that's perhaps more robust to hook changes.

Instead of using a hook, I just monkey-patched the SymlinkPlugin class to disable its "apply" function.

To do so, add this to the top of your webpack.config.js file:

//import SymlinkPlugin from "enhanced-resolve/lib/SymlinkPlugin";
const SymlinkPlugin = require("enhanced-resolve/lib/SymlinkPlugin");

SymlinkPlugin.prototype.apply = function () {
	//console.log('Symlink-plugin disabled...'); // uncomment to confirm disabled
};

Some other noteworthy additions/alternatives:

1. Resolve plugin (uses root when satisfies sem-ver): Unexpected behavior with linked NPM modules #985 (comment)
2. Resolve-config (symlinks: false, modules: root): Unexpected behavior with linked NPM modules #985 (comment)