This repository has been archived by the owner. It is now read-only.

npm link doesn't play nice with peer dependencies #5875

Closed
aldendaniels opened this Issue Aug 4, 2014 · 47 comments

Comments

Projects
None yet
@aldendaniels
Copy link

aldendaniels commented Aug 4, 2014

It's a simple use case:

  1. Break out shared helpers in a multi-project codebase
  2. Include helpers using npm link
  3. Helpers use Peer Dependencies (in my case because some helpers are also required via browserify, where it's important not to duplicate dependencies)

The gotcha is that when a library is included using npm link, npm doesn't appear to treat the linked directory as a child of the parent project, so peer dependencies aren't resolved.

@jaredhanson created this small library to work around precisely this issue.

Unless I'm missing something, npm's behavior in this scenario is non-ideal. Linked dependencies should be treated as if they are children (e.g. in the node_modules directory) of the project that includes them.

Thoughts?

@aldendaniels

This comment has been minimized.

Copy link
Author

aldendaniels commented Aug 5, 2014

For the time being, I'm working around this issue by putting the following at the top of my helper JS files:

// Force Node to load modules from linking parent.
require = (typeof window === 'undefined' ? require('parent-require') : require);

Browserify correctly handles linked modules, so I only need this hack for node.


On second thought, perhaps I should have posted this issue as NodeJS issue rather than NPM as the issue has to do with how require() calls are resolved rather than anything NPM specific.

@othiym23

This comment has been minimized.

Copy link
Contributor

othiym23 commented Aug 5, 2014

Adding this to the bucket of many, many confusing edge cases that need to be resolved when we get around to nailing #5080 to the floor (which is on the roadmap to happen later this year). Thanks for the report, and thanks for the example workaround. I think we're going to have to look at all the stuff going on with peerDependencies before we can figure out the simplest, most consistent way to handle all of this.

@othiym23

This comment has been minimized.

Copy link
Contributor

othiym23 commented Aug 5, 2014

I do think you're right that require()'s behavior is unhelpful in this case, but I also think that given how important having that behavior locked down is to the stability of the whole Node ecosystem, it's going to be very tough to change now.

@aldendaniels

This comment has been minimized.

Copy link
Author

aldendaniels commented Aug 5, 2014

Thanks for the update - makes sense.

@othiym23 othiym23 added the bug label Sep 21, 2014

@appsforartists

This comment has been minimized.

Copy link

appsforartists commented Oct 8, 2014

I've probably spent 3 - 4 days trying to workaround a similar issue before I realized it was a bug in NPM. It didn't make sense that you couldn't require() something declared in peerDependencies, but I wasn't able to. Instead, I tried all sorts of crazy shit (including forcing the calling app to pass in a path to its node_modules to generate my own absolute paths in my library's require statements, and having the calling app pass in a reference to its own module so I could try parentModule.require):

var { LibraryName } = require("libraryName").forModule(module);

I ended up moving my library repo inside my app repo's node_modules folder. It's a scary workaround, because it would be way too easy to accidentally clobber the whole repo when trying to other things with npm, but it makes the paths resolve correctly.

appsforartists added a commit to appsforartists/react-router that referenced this issue Oct 8, 2014

[fixed] React dependency
It should be in peerDependencies, not devDependencies.  Otherwise,
you'll get needless failures on `npm list`.

Note: if you are developing ReactRouter itself, your repo should be
inside your calling app's node_modules until this is fixed:

npm/npm#5875

If you try to instead resolve this by installing a copy of React inside
ReactRouter's node_modules, you're liable to end up with a broken
build, because webpack will include both copies in parallel.

appsforartists added a commit to appsforartists/react-router that referenced this issue Oct 8, 2014

[fixed] React dependency
It should be in peerDependencies, not devDependencies.  Otherwise,
you'll get needless failures on `npm list`.

Note: if you are developing ReactRouter itself, your repo should be
inside your calling app's node_modules until this is fixed:

npm/npm#5875

If you try to instead resolve this by installing a copy of React inside
ReactRouter's node_modules, you're liable to end up with a broken
build, because webpack will include both copies in parallel.

appsforartists added a commit to appsforartists/ambidex that referenced this issue Dec 24, 2014

Removing ambidex.forModule and <AmbidexRoutes>
Most of the clever stuff I've been doing in this repo has been trying to
work around [npm#5875](npm/npm#5875).  Now that I realize this is a legitimate bug in
npm (with a simple workaround - develop your library in its calling
app's node_modules folder), I can remove all the clever shit and let
Ambidex shine through.
@seeden

This comment has been minimized.

Copy link

seeden commented Mar 8, 2015

+1 Any update?

@yoavniran

This comment has been minimized.

Copy link

yoavniran commented Mar 20, 2015

ran into this issue now with a helper im writing (used with npm link) not working because it cant require the dependency from the parent.
Would love to get an update on whether this will be fixed anytime soon.

@ssbb

This comment has been minimized.

Copy link

ssbb commented Apr 8, 2015

+1 any update?

@Dakuan

This comment has been minimized.

Copy link

Dakuan commented Apr 21, 2015

+1

1 similar comment
@francoislaberge

This comment has been minimized.

Copy link

francoislaberge commented May 18, 2015

+1

@mciparelli

This comment has been minimized.

Copy link

mciparelli commented May 19, 2015

+1. This would be nice to have!

@henrikhaugboelle

This comment has been minimized.

Copy link

henrikhaugboelle commented May 26, 2015

+1

@barneycarroll

This comment has been minimized.

Copy link

barneycarroll commented Jun 19, 2015

Browserify correctly handles linked modules, so I only need this hack for node

@aldendaniels I haven't found that to be the case at all. Hours of head-scratching have finally led me to understand that my browser is loading 2 instances of framework X, whose singular runtime is essential for my application to work with the plugin I'm attempting to debug. Any magic at all involved in your Browserify solution?

@doctariouz

This comment has been minimized.

Copy link

doctariouz commented Jun 25, 2015

+1

@aldendaniels

This comment has been minimized.

Copy link
Author

aldendaniels commented Jul 12, 2015

@barneycarroll - sorry for the delayed reply. No, no magic that I remember. I'm now using NPM private modules instead of the structure described in this issue, so this is no longer affecting me.

The issue that I encountered when I opened this was independent from the module duplication problem (which gets better in NPMv3, BTW). The issue was that NPM would climb the directory from a linked module when resolving peerDependencies instead looking up via the symlink.

I don't know whether or not this behavior still exists today.

@francoislaberge

This comment has been minimized.

Copy link

francoislaberge commented Jul 12, 2015

Yeah, hopefully this becomes less of an issue as it becomes easier to dedupe a module in v3:
http://www.infoq.com/news/2015/06/npm

@donabrams

This comment has been minimized.

Copy link

donabrams commented Jul 20, 2015

Oi, the normal workaround doesn't work with use of require('babel-core/register') Why does npm link $LIB install peer dependencies instead of walking the tree? If it's too late to change can we at least add a flag --no-peer to npm link?

@donabrams

This comment has been minimized.

Copy link

donabrams commented Jul 20, 2015

Actually this is fixed in npm3, no need to worry.

@vjpr

This comment has been minimized.

Copy link

vjpr commented Aug 6, 2015

I'm having this problem too. It seems a simple fix would be to walk up the dir structure from the symlink.

E.g.

cd ~/dev/foo
npm link ~/dev/bar
ls -al ~/dev/foo/node_modules
bar -> ~/dev/bar

If we look at require.cache['~/dev/bar/index.js'].paths it would be:

['~/dev/bar/node_modules', '~/dev/node_modules', '~/node_modules', '/Users/node_modules', '/node_modules']

So walking up the symlink would fix it. Didn't appear to be the case in npm3.

Alternatively, you could create a directory such as ~/dev/foo/local_modules/bar, and npm link ./local_modules/bar, which would cause ~/dev/foo/node_modules to be one of the stops in the require search path walk.

The downside is you cannot share this library with multiple projects easily - you would have to run npm link each time, and npm link is designed to be shared globally.


Another idea is the add all the linked modules to a dir somewhere in the file system, then create a symlink to your current project's node_modules. This symlink would have to be adjusted for each project you work on.

E.g.

ls ~/dev/linked-modules
bar
node_modules -> ~/dev/foo/node_modules

ls ~/dev/foo
ls -al node_modules
bar -> ~/dev/linked-modules/bar
@louy

This comment has been minimized.

Copy link

louy commented Aug 13, 2015

I had the same issue while working with a framework that used React. I ended up with two versions of react which gave me very weird errors.

I worked around this issue by adding a file within that framework called react.js which is basically the following:

module.exports = require('react');

and in my application, I required that file instead of the original react.

Of course, this is an inconvenient workaround; there was a bug that I wanted to fix and I only had this option.

@iarna

This comment has been minimized.

Copy link
Member

iarna commented Sep 16, 2015

So the work around for this in npm@2 would be to directly depend on the various peerDependencies. In npm@3, peerDependencies never trigger installs, so what was a work around in npm@2 is now mandatory behavior for npm@3.

As I don't believe this issue makes sense any more in the context of npm@3, I'm going to close it, but if you feel otherwise or have some other related problem, please don't hesitate to open a new issue.

@iarna iarna closed this Sep 16, 2015

@VanCoding

This comment has been minimized.

Copy link

VanCoding commented Sep 25, 2015

Hi guys, Im on npm 3.3.3 and node 4.1.1 and still have problems with this!

I'm working on 2 packages simultaneously: app and plugin. app depends on plugin and react, while plugin peer depends on react.

Since it's impossible to place plugin into the node_modules directory of app, I've done npm link in plugin and then npm link plugin in app.

Now when I try to start, plugin cannot find react, even if it is correctly located in the node_modules directory of app. Interestingly, when I console.log(__dirname) in plugin's index.js, then it returns it's real path rather than its virtual path.

So if I have the following folder structure:

/path/to/modules/
    app
        index.js
        node_modules
            react
                ...
            plugin -> app
    plugin
        index.js

Then it outputs
/path/to/modules/plugin rather than /path/to/modules/app/node_modules/plugin which I think is the reason why it cannot find react.

What am I doing wrong here?

@yordis

This comment has been minimized.

Copy link

yordis commented Nov 30, 2015

👍

@cztomsik

This comment has been minimized.

Copy link

cztomsik commented Dec 1, 2015

You can use require.main.require('peerPkgName')
http://stackoverflow.com/a/25800501/569051

I use it and it works quite good.

@raine

This comment has been minimized.

Copy link

raine commented Feb 16, 2016

Great workaround, @tomwidmer.

I have this problem with hubot when using npm link with an adapter you're developing. The adapter module does not have hubot as a dependency of its own but still expects require('hubot') to work and load it from the parent. If you try to install hubot as its own dependency under ./node_modules/hubot-gitter2/node_modules/hubot, "module not found" error will go away, but weird things happen and things don't work as expected.

@dalanmiller

This comment has been minimized.

Copy link

dalanmiller commented Mar 8, 2016

Still having this issue on npm 3.6.0. This process should be a lot easier.

@artisologic

This comment has been minimized.

Copy link

artisologic commented Mar 16, 2016

Hi,

I'm still having the same issue as @VanCoding as well.
Pretty much same situation with react.

I have a main package that depends directly on react and other-package.
Then other-package has a peer dependency of react.

Did npm link and I also get Cannot resolve module 'react'

I don't think this issue has been resolved, is there a new one opened for this?

@VanCoding

This comment has been minimized.

Copy link

VanCoding commented Mar 16, 2016

@artisologic The real problem we're having here is this: nodejs/node#3402

npm link is completely useless until node.js changes the behavior of require. I'm still stuck to npm2 because of this.

@rstacruz rstacruz referenced this issue Apr 25, 2016

Closed

Global store #19

@dlongley

This comment has been minimized.

Copy link

dlongley commented Apr 27, 2016

If you're using node 6.x, symlinks are now preserved (not dereferenced with realpath), so npm link may now play a little more nicely with peer dependencies.

@kobalicek

This comment has been minimized.

Copy link

kobalicek commented May 23, 2016

I'm on node 6.2 and it's still broken - I cannot develop packages that have peer dependencies

@dlongley

This comment has been minimized.

Copy link

dlongley commented May 23, 2016

@kobalicek -- That's because they just rolled back the change on node 6.2. It's currently hidden behind a flag --preserve-symlinks (see: nodejs/node#6537) until the behavior can be integrated back in without causing a problem for a few other use cases. Using that flag, you should be able to develop peer dependencies.

@kobalicek

This comment has been minimized.

Copy link

kobalicek commented May 23, 2016

Is there anywhere an explanation why it doesn't work and what's the technical reason behind it (also node bug would help)? Because I think this is a valid use case that should just work without any flag.

Suppose I have two packages a and b, where b requires a and I have a directory structure like this:

main_package
  node_modules
    a -> symlink
    b -> symlink

Then b fails to require a if b is a symlink, but it works if b is a directory. This means that node doesn't walk the file-system back from b to the main_package, so where it walks?

@dlongley

This comment has been minimized.

Copy link

dlongley commented May 23, 2016

@kobalicek,

Yes, I meant to link to this issue: nodejs/node#6537

The original issue is here: nodejs/node#3402

And the original fix is here: nodejs/node#5950

hhjcz added a commit to hhjcz/too-simple-react-skeleton that referenced this issue Sep 1, 2016

@bholloway

This comment has been minimized.

Copy link

bholloway commented Sep 28, 2016

When npm@3 was released I read somewhere that peerDepdendencies are deprecated and should no longer be used.

Which seems to be confirmed by this comment.

As I understand it, you should convert peerDependencies to dependencies.

  • npm@3 installs maximally flat. Both your package and the containing project specify the dependency. But it will only be installed in the containing project.
  • When you npm link your package it is equivalent to running npm install in the linked directory. It does not see the containing project and cannot leverage the existing dependency installation there. The dependency is installed in both the containing project and the linked directory.

Isn't this the desired behaviour?

Personally I am confused because I see lots of react code that is still using peerDependencies and people with projects broken by npm link.

That said, I still experience problems with libraries that use Singleton Symbol constants:

Consider a library that stores a value using a Symbol key. Data is stored in a linked project and accessed in the containing project. However, these both have different installations of the package and so different instances of the Symbol.

Maybe this issue is solved?

@barneycarroll

This comment has been minimized.

Copy link

barneycarroll commented Sep 28, 2016

I read somewhere that peerDepdendencies are deprecated and should no longer be used.

Which seems to be confirmed by this comment.

It's not deprecated: the implementation has changed. It's explained in the package.json docs: peerDependencies will never be installed automatically as a result of installing a dependency that specifies a peer dependency you don't have. Instead you will get a warning about the need for that dependency.

So if you have dependent package A#1, and install dependent package B#1 which specifies A#1 as a peer dependency, you're fine. If you didn't have A#1, the install procedure will ask you to resolve this yourself.

This puts the onus on the author to use their own judgment on a case by case basis, because there is no procedural correct solution for eg you are dependent on package A#1 and install a package B#2 that has a peer dependency on A#2.

@bholloway

This comment has been minimized.

Copy link

bholloway commented Sep 28, 2016

@barneycarroll you explain the problem well.

I guess I am arguing that the author of B#1 should change A#1 from peerDependencies to dependencies. Since npm@3 de-duplication gives the desired outcome.

The exceptions is if the A#1 you already have is semver incompatible, in which case you will get a silent duplication. Which may be a deal-breaker. Do you see any other down sides?

The other issue is that some people may still be using npm@<3.

@abelnation

This comment has been minimized.

Copy link

abelnation commented Oct 6, 2016

When you npm link your package it is equivalent to running npm install in the linked directory. It does not see the containing project and cannot leverage the existing dependency installation there. The dependency is installed in both the containing project and the linked directory.

@bholloway have you figured out a way to address the other issue you mentioned in your above comment? Would npm dedupe be enough to remove the duplicated deps from the linked sub-package?

@bholloway

This comment has been minimized.

Copy link

bholloway commented Oct 13, 2016

@abelnation I have not tried it. If it does de-duplicate modules as you hope then it will break your code unless you can use the --preserve-symlinks flag.

@dcaillibaud

This comment has been minimized.

Copy link

dcaillibaud commented Nov 16, 2016

More than two years later the pb is still here. I don't want to duplicate eslint, babel and their plugins and preset in every project (more than 300k files each time, I run out free inodes !), so I link them, but with eslint it doesn't work, I still have Cannot find module 'eslint-config-standard'

Changing shebang in /usr/lib/node_modules/eslint/bin/eslint.js from #!/usr/bin/env node to #!/usr/bin/nodejs --preserve-symlinks doesn't solve the pb (npm 3.10.9 and node 6.9.1), so the only workaround I found is making hard links on eslint package (others eslint-* can play whith npm link), it preserve a little space but not inodes :-/ (hopefully it isn't the biggest package)

@cztomsik

This comment has been minimized.

Copy link

cztomsik commented Nov 16, 2016

@dcaillibaud require.main.require('pkg') does not work for you?

@dcaillibaud

This comment has been minimized.

Copy link

dcaillibaud commented Nov 16, 2016

@cztomsik I don't understand, I don't want to modify the eslint module to use it if it's linked, same for babel, webpack & co. You suggest to make wy own eslint wrapper, and use it for linting ?

@kevzettler

This comment has been minimized.

Copy link

kevzettler commented May 8, 2017

You can npm link the peer dependencies back to your host projects dependencies

http://stackoverflow.com/a/38818358/93212

@nelsonomuto

This comment has been minimized.

Copy link

nelsonomuto commented Aug 4, 2017

can't you install the peer dependency as a dev dependency?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.