Skip to content
This repository has been archived by the owner. It is now read-only.

`npm install` after `npm link` will "steal" dependencies from linked packages #10343

Closed
jhenninger opened this issue Nov 11, 2015 · 90 comments
Closed

Comments

@jhenninger
Copy link

@jhenninger jhenninger commented Nov 11, 2015

Create two packages:

jo@t420:/tmp/npmtest$ mkdir A B

A depends on lodash:

jo@t420:/tmp/npmtest$ cd A
jo@t420:/tmp/npmtest/A$ npm init --yes && npm install --save lodash
Wrote to /tmp/npmtest/A/package.json:

{
  "name": "A",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


A@1.0.0 /tmp/npmtest/A
`-- lodash@3.10.1 

npm WARN EPACKAGEJSON A@1.0.0 No description
npm WARN EPACKAGEJSON A@1.0.0 No repository field.

B depends on A which is symlinked for development:

jo@t420:/tmp/npmtest/A$ cd ../B
jo@t420:/tmp/npmtest/B$ npm init --yes                       
Wrote to /tmp/npmtest/B/package.json:

{
  "name": "B",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Create the link:

jo@t420:/tmp/npmtest/B$ npm link ../A
npm WARN EPACKAGEJSON A@1.0.0 No description
npm WARN EPACKAGEJSON A@1.0.0 No repository field.
/home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A -> /tmp/npmtest/A
/tmp/npmtest/B/node_modules/A -> /home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A -> /tmp/npmtest/A

So far so good, A still has lodash in its node_modules:

jo@t420:/tmp/npmtest/B$ tree -d ..
..
├── A
│   └── node_modules
│       └── lodash
│           ├── array
│           ├── chain
│           ├── collection
│           ├── date
│           ├── function
│           ├── internal
│           ├── lang
│           ├── math
│           ├── number
│           ├── object
│           ├── string
│           └── utility
└── B
    └── node_modules
        └── A -> /home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A

18 directories

Now actually save A as a dependency of B in package.json:

jo@t420:/tmp/npmtest/B$ npm install --save ../A
npm WARN update-linked node_modules/A needs updating to 1.0.0 from 1.0.0 but we can't, as it's a symlink
lodash@3.10.1 node_modules/A/node_modules/lodash -> node_modules/lodash
npm WARN EPACKAGEJSON B@1.0.0 No description
npm WARN EPACKAGEJSON B@1.0.0 No repository field.

And all dependencies of A get moved to B's node_modules folder:

jo@t420:/tmp/npmtest/B$ tree -d ..
..
├── A
└── B
    └── node_modules
        ├── A -> /home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A
        └── lodash
            ├── array
            ├── chain
            ├── collection
            ├── date
            ├── function
            ├── internal
            ├── lang
            ├── math
            ├── number
            ├── object
            ├── string
            └── utility

17 directories

Running tests etc. in A will now fail due to missing dependencies, unless you run
npm install in A again.

This will also happen when manually adding A as a dependency to B's package.json
and then running npm install.

npm version: 3.3.12

@iarna
Copy link
Contributor

@iarna iarna commented Nov 11, 2015

So, I think this is caused by two things:

  1. 33aedaf which caused npm to move stuff around when it shouldn't.

AND

  1. A specific bug that allows moves to happen from inside linked modules. (This https://github.com/npm/npm/blob/v3.4.0/lib/install/diff-trees.js#L145-L152 should not generate moves if the source is inside a link. But further, https://github.com/npm/npm/blob/v3.4.0/lib/install/diff-trees.js#L119-L126 probably shouldn't generate removes if the source is inside a link. Possibly these should be like update-link, which is to say, a different action that just warns.)
i-e-b added a commit to i-e-b/npm-workspace that referenced this issue Nov 13, 2015
npm v3 has broken a few behaviours that `npm-workspace` relies on.

- Peer dependencies:
  These are no longer installed by default. When `npm-workspace`
  explicitly adds these, they remain functional. Test have been updated to
  reflect new behaviour.

- Linked dependency flattening ( npm/npm#10343
  and npm/npm#10348 and
  npm/npm#9999 )
  npm v3 does not correctly install dependencies that are shared between a
  package and its sym-linked dependency.
```
  A
  +- B (linked)
  |  +- C
  +- C
```
  Should result in both `A` and `B` getting a copy of `C`, however it
  results in:
```
  A
  +- B (linked)
     +- C
```
  To work around this, npm-workspace now adds 'dummy' packages for ones it
  will link, then runs `npm i` then replaces the dummy packages with the
  real symlinks. This results in a complete and correct install.
  Due to a change in the way npm checks versions after npm 3, the dummy
  packages need to be different, so the npm version is read at the start
  of an install.
@baelter
Copy link

@baelter baelter commented Nov 25, 2015

+1 this is burning time = money and sanity. Target release? Need help?

@ElliotChong
Copy link

@ElliotChong ElliotChong commented Dec 9, 2015

Ghetto hacky workaround idea that might help the next developer: Try using the NODE_PATH env variable to treat B as a node_modules path.

NODE_PATH=/path/to/B/node_modules node index.js

https://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

@timdp
Copy link

@timdp timdp commented Dec 23, 2015

I've had some success working around this issue by rebuilding my node_modules folder with this script (written for Windows). However, it often triggers "refusing to delete: containing path isn't under npm's control" errors. I'm sure that's a different issue underneath. Regardless, npm link is just broken at this point.

@whitecolor
Copy link

@whitecolor whitecolor commented Dec 30, 2015

+1 when it is going to be fixed?

@tehsenaus
Copy link

@tehsenaus tehsenaus commented Jan 22, 2016

Any update on this?

@whitecolor
Copy link

@whitecolor whitecolor commented Jan 22, 2016

That actually want holding many from using npm3.

@baelter
Copy link

@baelter baelter commented Jan 26, 2016

@whitecolor Maybe not but it's still an undesirable "feature"

@baelter
Copy link

@baelter baelter commented Feb 10, 2016

I think the whole link command seems a bit over complicated maybe? Correct me if I'm wrong but wouldn't the following do what we want @iarna ?

npm link x should only do:

  1. force move node_modules/x to node_modules/.x
  2. create symlink

npm unlink [x] should only do:

  1. If no argument, do for all symlinks, else do for x:
  2. remove symlink
  3. move node_modules/.x to node_modules/x

Then you should probably carry on with an npm update, but that should maybe not be a part of linking.

Ps. We should also have a npm list links

@larsenjh
Copy link

@larsenjh larsenjh commented Feb 18, 2016

+1 please fix this. We have a very complex module structure, and this problem causes many of us to have to re-install all our deps in all our projects over again. Terrible bug.

@timdp
Copy link

@timdp timdp commented Feb 18, 2016

@baelter As I understand it, npm link itself is fine, but npm install doesn't treat links any differently (or maybe it does, but badly), which causes inconsistencies.

@baelter
Copy link

@baelter baelter commented Feb 18, 2016

@timdp It looks to me like npm link x does a lot more that what I suggested above. npm link is simple enough and I haven't found any issues with that.

@schmod
Copy link
Contributor

@schmod schmod commented Mar 25, 2016

@timdp Agreed, and I'm not sure whether or not this is intentional. If nothing else, it's a significant departure from npm2.x's behavior.

For example, if I have smallApp that depends on bigLib, I might do something like this:

$ cd bigLib
$ npm install
$ npm link

$ cd ../smallApp
$ npm link bigLib
   ~/smallApp/node_modules/bigLib -> ~/npm/lib/node_modules/bigLib -> ~/bigLib
$ npm install 

If smallApp has no other dependencies, I would expect that final npm install command to do nothing, because all of bigLib's dependencies are already installed in bigLib/node_modules. (This what npm2.x would have done)

Instead, running npm install from smallApp installs a duplicate set of bigLib's dependencies into smallApp's node_modules folder. Because bigLib is big and has many dependencies, this could potentially be a very lengthy operation.


Ideally, npm3 should avoid tree-flattening when traversing into linked packages. I'm having a difficult time envisioning a situation where the current behavior is desirable.


One tricky case worth considering:

If bigLib is missing a dependency, should npm install a copy of the missing dep into smallApp?

Or can we enforce a firm contract that npm does not resolve transitive dependencies for linked modules, thus making the linked module completely responsible for providing its dependencies?)

@i-e-b
Copy link

@i-e-b i-e-b commented Mar 25, 2016

@schmod

thus making the linked module completely responsible for providing its dependencies?

I would say this is the most sensible option, both closest to npm 2, and it supports the case where a linked dependency may be part of a wider project and is being included in various places for packaging reasons.

@timdp
Copy link

@timdp timdp commented Mar 27, 2016

I'd just always install/keep all the linked package's dependencies inside its node_modules, regardless of whether you ran npm install for the package itself or for one that links to it. That way, at worst, you'll end up with some duplicate dependencies caused by changes to package.json, which should be easy to fix with npm prune. At the same time, all your packages will have all their dependencies available, no matter how they're getting loaded. But I sense I'm missing something, since I don't understand why this continues to be an issue in npm 3.

@ghost
Copy link

@ghost ghost commented Apr 9, 2016

endless frustration with this issue, endless.

@othiym23 othiym23 modified the milestone: understandable links Apr 13, 2016
asfgit pushed a commit to apache/cordova-lib that referenced this issue May 11, 2016
 This closes #437
@rochoa rochoa mentioned this issue May 23, 2016
schmod pushed a commit to schmod/npm that referenced this issue May 24, 2016
Prevents npm install from crossing into the symlinked package, and
trying to reinstall all of its dependencies

Mitigates npm#10343
@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 26, 2017

@alan-agius4 I tested with the latest version of npm (5.0.3).

@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 26, 2017

So I think @basicdays may have discovered the fix they implemented here and this issue was simply left open/forgotten.

I just decided to go back and repro the original issue. First I did it with npm version 4.2.0.

$ mkdir A B
$ cd A
$ npm init --yes
$ npm install --save lodash
$ cd ../B
$ npm init --yes
$ npm link ../A
$ npm install --save ../A

npm4-link-issue

This correctly reproduced the issue. Lodash disappeared from A's node_modules and appeared in B's node_modules. Notice that the reproduction steps link to the module and then install the module from the local file system.

I verified that in npm 4 it did indeed copy the module from the referenced path into B's node_modules folder. It did not create a symlink, hence the need for npm link in addition to local file system npm install.

I then went forward and tried the reproduction steps in npm@5 and got some interesting results.

npm5-link-issue

Notice that when I did npm install ../A portion after doing npm link ../A it just nuked A's node_modules folder. I'm not sure why that happened but B's package.json file did get updated with a reference to A on the file system so I decided to just run npm install after that. Suddenly the tree looked normal with lodash living in A's node_modules folder as expected.

What I am taking away from the above is that npm link felt redundant because you can just do this:

npm5-link-issue-solved

One thing I've yet to do is test npm link with the package installed from the registry instead of the file system. I'll do that next. Installing from file system now seems to create a symlink whereas in npm@4 it actually copied the files, making it a better simulation of installing from the npm registry. Since it now creates a symlink the original reproduction steps don't apply. For now it looks like @basicdays has figured out a hell of a workaround (or possible official solution that nobody has updated us about). I'll keep testing around but this behavior hints to me that it's intentional.

@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 26, 2017

I can't reproduce the original bug of the linked project's dependencies being flattened into the parent project's dependencies with npm@5, not even with a library that is on the registry. So the original bug isn't an issue anymore. Here I'll go through the scenarios I set up and why I think this new behavior is intentional. Whether or not we users should be happy with it or not is up for discussion.

Here are the following scenarios I've tested with a module that lives on the registry and on my machine somewhere (remember that --save is implied in npm@5+):

Just a quick note for anyone confused about npm link:

cd app && npm link path/to/dependency
is equivalent to
cd path/to/dependency && npm link
followed by
cd app && npm link dependency-name

Both ways create a symlink from the dependency to npm's global node_modules and then a symlink from the app to that global symlink.

Creating link after install.

$ mkdir app && cd app
$ npm init -y
$ npm install registry-package
$ npm link local/path/to/registry-package

In this scenario the installed registry package is overwritten by the symlink. Any installed modules in the local symlinked package seem untouched 🎉

Installing package after linking it.

$ mkdir app && cd app
$ npm init -y
$ npm link local/path/to/registry-package

This so far correctly links to the local package and that package's dependencies are untouched 🎉

But then if you:

$ npm install registry-package

It will overwrite the symlink with the actual package from the registry. However there is a weird side effect here. The previously linked package who's symlink just got overwritten also gets its node_modules directory nuked from existence for seemingly no reason 😠

What I am taking away from this is that npm link no longer seems intended to be used to create a permanent local link. It's more for a quick link for debugging. For example, if you're having problems in your app and you've traced it to a dependency then you can quickly npm link path/to/dependency and you're good to debug real quick without affecting your app's package.json reference to the actual registry package.

If you want a permanent link to a local package, say for an app still in active development, you would run npm install path/to/dependency which permanently symlinks the package into your app with a "dependency": "file:path/to/dependency" reference in your package.json. Then when you're ready to go live you would publish your dependencies and replace those references in your app with the actual registry packages.

This is all poorly documented and nobody has stopped by this issue to update us, but this behavior seems deliberate to me. I think this particular issue can be closed. Though I might want to open a new issue for that odd side effect where installing over an existing symlink gives the symlinked package a courtesy wipe of its node_modules for no reason.

@dcaillibaud
Copy link

@dcaillibaud dcaillibaud commented Jun 27, 2017

Here is an history of this bug.

  • npm2 : all is fine and work as expected (npm link create symlink and npm install doesn't change anything in linked packages)
  • npm3 & npm4 : npm link create symlink, but npm install move dependencies of the linked package under the main project => linked package himself is broken, you need to re-run npm install in it
  • npm5 : same bug as npm3 & 4, but it's worse as npm install destroy all symlinks

My need is to work with a modular approach, here is a sample hierarchy

  • projectA
    -- myModule1
    --- myModule2
    -- myModule3
    --- myModule2
    -- projectB (can be run as standalone app or embedded app)
    --- myModule4

  • projectC
    -- myModule1
    --- myModule2
    -- myModule4

with npm2, npm link create symlink and I can have all my code at only one place on my disk, so during development changing something in a project dependency change it everywhere, if it broke something I see it immediatly (as long as a test exists and is running in background), without pushing it on a repo increasing version, re-install it on another project changing the project package.json and seeing then something missing.

with npm3 & 4, I need to re-run npm install in all my own dependencies after doing it in a project.

npm5 is clearly unusable in this case.

I don't want to use lib approach with link in the package.json, because I want each module to have it's own repo, and no link in production.

@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 27, 2017

I'm not saying it's right or wrong the way it is now, especially since all we have is radio silence. But I'm not sure I understand why each module can't have it's own repo in the setup I demonstrated above? Personally I kind of like that I can npm install ./path/to/myModule and get a permanent symlink because it allows me to nuke my app's node_modules and run npm install and have full confidence my symlinks will return, with their own dependencies intact as well. In npm 4 and lower running npm install ./path/to/myModule did not create a symlink but it does in 5.

My only worry is that I might accidentally commit that reference one day and break my app's build. If that's all you mean then I agree that's a potential nuisance. I'd almost prefer to have a separate file that gets used as an override during npm install and that file could contain the symlink references but be ignored in source control.

I do however see potential with this if I were to say, swap those references out for the real thing in a build script. Then in dev I could have permanent symlinks that I never have to restore, even if I totally delete node_modules in my app. It seems to me that hierarchy you described is definitely doable in npm@5 with the only downside being that your package.json would have "yourModule": "file:./path/to/yourModule" and you'd have to replace those with actual registry references before going to staging/production.

For example, you could potentially create something like a sym-map.json file as a manifest of what to swap out when it's time to go live.

// projectA/package.json

{
  ...
  "dependencies": {
    "myModule1": "file:../myModule1",
    "myModule3": "file:../myModule3",
    "projectB": "file:../projectB"
  }
}
// projectA/sym-map.json

{
  "myModule1": "^3.9.1",
  "myModule3": "^1.2.1",
  "projectB": "^4.0.3"
}

Then just have your build script reference that file during build and swap them out.

var manifest = require('package.json');
var symMap = require('sym-map.json');
var fs = require('fs');

Object.keys(symMap).forEach((name) => {
  manifest.dependencies[name] = symMap[name];
});

fs.writeFile('./package.json', JSON.stringify(manifest));

You could even turn that into a script you could run at any time to swap back and forth.

// swap-references.js

var manifest = require('package.json');
var symMap = require('sym-map.json');
var fs = require('fs');

Object.keys(symMap).forEach((name) => {
  let manifestValue = manifest.dependencies[name];
  manifest.dependencies[name] = symMap[name];
  symMap[name] = manifestValue;
});

fs.writeFile('./package.json', JSON.stringify(manifest));
fs.writeFile('./sym-map.json', JSON.stringify(symMap));

Then run node swap-references.js to switch back and forth at will.

It's a little more work but I think the package.json reference might actually provide more utility by making your symlinks more permanent while in dev. It's more explicit rather than expecting npm install to detect and skip over symlinked modules in node_modules. Not to mention that in the old way if you nuked node_modules in your app then you'd have to do the npm link yourModule all over again.

Anyway, just an idea. Not saying you're wrong at all. I just want to understand your needs better to see if there is indeed no way to accomplish what you're after with npm@5.

@laat
Copy link

@laat laat commented Jun 27, 2017

EDIT: Nevermind, I cannot make this work with deeper symlink chains pkg1 -> pkg2 -> pkg3
Ignore this comment.

This seems to work in my tests, as it is not breaking symlinks or steals dependencies

$ npm -v
5.0.4

$ npm link other
$ npm install --preserve-symlinks --no-shrinkwrap

But, as in the original bug report this still steals dependencies:
(the flags do not seem to matter)

$ npm link other
$ npm install --preserve-symlinks --no-shrinkwrap ../other

I would love to see this issue fixed, preserving the symlink behaviour from npm@2 on npm install

@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 27, 2017

@laat there's a few issues with your examples.

  1. Those flags aren't causing the difference between your two scenarios (and I'm pretty sure npm has no --preserve-symlinks flag as that is a flag you pass to node, not npm.)

  2. npm link does not modify package.json. So running npm link other and then npm install has no conflicts because npm link didn't add anything to package.json and npm install by itself just reads from package.json to perform installs.

  3. Your second scenario is exactly what I described above with npm 5. If you do npm link other and then npm install ../other the symlink created by npm link other gets overwritten by a symlink created by npm install ../other. Npm 4 and earlier did not create symlinks if you did npm install ../some/path but in npm 5 it does create one, making npm link redundant. The difference is that npm install ../other does modify package.json and adds this explicit reference:

    "dependencies": {
      "other": "file:../other"
    }
    

    With that reference in package.json npm install will now always create a symlink to that file path, even if you delete node_modules altogether and run npm install fresh.

  4. Your second scenario doesn't actually "steal" dependencies from "other". There's an odd side effect when you use npm install ../other to overwrite a symlink created by npm link other where it straight up deletes the node_modules directory from "other".

Basically I believe they've changed how they expect us to use npm link and just haven't updated us here or the documentation.

  • If you want a temporary link to "other" then you run npm link other and overwrite the installed module with a symlink. Running npm install after that will overwrite the symlink with the actual package from the registry again. npm link is not intended to create a permanent link anymore.

  • If you want a permanent link to "other" then you run only npm install ../other without using npm link at all. It adds a reference to package.json and creates a permanent symlink to "other" that won't be overwritten unless you change the reference in package.json to point to the registry. This allows you to run npm install as much as you want and it won't affect that permanent link to "other". To change it to point to an actual package you would run npm install other changing package.json from "other": "file:../other" to the traditional format "other": "^1.2.3". which points to the registry.

@dcaillibaud
Copy link

@dcaillibaud dcaillibaud commented Jun 27, 2017

  1. I want permanent link in my dev env, because my apps and their dependencies are in a living stage, not just today. having to run npm link on every dependency of my own is really painfull, and lead to a lot of duplicates (dependencies of dependencies, placed at the root level by the npm install)
  2. I don't want to put "dependency": "file:path/to/dependency" in my package.json because path/to/dependency is specific on my local machine, others people coding on this app have others path (or none, if they don't touch dependencies code), and in production I don't want any link.
@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 27, 2017

@dcaillibaud Right, I think I understand. You don't want to check in "dependency": "file:path/to/dependency". That's the one thing I agree is an added nuisance, but there are ways around it like I described above with my little script to swap them out at build/publish time.

What I wish they had done instead is allow us to create some kind of local-dependencies.json file and store those symlink references in there instead of package.json. And have npm install respect that file as an override for dependencies listed in package.json. Then you could just add local-dependencies.json to .gitignore and not have to modify package.json at all. That'd be the best of both worlds. You could totally delete node_modues and run npm install fresh locally and your symlinks would be restored (something you couldn't even do in npm 2 without having to redo all your links) and package.json wouldn't have to have hard references that need changed before being committed or published to production.

@dcaillibaud
Copy link

@dcaillibaud dcaillibaud commented Jun 30, 2017

@chevex It would be great if npm could manage local overrides through e.g. --local-overrides package.local.json

Other solution could be to pass the package.json content to the preinstall script (and be aware of changes), because I tried to modify package.json file at preinstall runtime, but npm doesn't re-read the file before installing.

So, I tried your solution with this script (github don't take js extension, so I put txt), but the syntax `"myModule": "file:../myModule" seems to copy files, not link directory !

This proposition could solve the pb, but in a partial way, still need a script in replacement of npm install, or one of the above suggestion to put this script in preinstall.

@dcaillibaud
Copy link

@dcaillibaud dcaillibaud commented Jun 30, 2017

I made a script which replace npm install

  • merge package.local-extras.json into package.json, stripping all "file:" dependencies (I don't want a copy)
  • run npm install
  • link all local "file:" dependencies
  • restore original package.json

Use it at your own risk !
https://github.com/Sesamath/sesatheque/blob/0.9.74/scripts/npmi.js

@chevtek
Copy link
Contributor

@chevtek chevtek commented Jun 30, 2017

You say it copied the files? You sure you were using npm 5? For me that file prefix always creates a Symlink to the directory.

@dcaillibaud
Copy link

@dcaillibaud dcaillibaud commented Jun 30, 2017

I saw this with npm@5.0.3, I'm pretty sure to have seen plain directories after a rm -rf node_modules; npm i (with some file: dependencies), but I can't reproduce it.

I just upgraded to 5.0.4, and I don't really understand, after erasing ~/.npm, putting my file:… in package.json, npm install doesn't install any of the file:… dependencies. If y specify explicitly one of my own module with npm install myPackage, the symlink is created. Then a rm -rf node_modules; npm i install registry modules and myPackage is linked, but other packages with file: aren't ! Like if you need to first install once the local module alone to force npm realize that the path is a real one…
It's the same after a revert to 5.0.3

@Madd0g
Copy link

@Madd0g Madd0g commented Jul 2, 2017

Just "upgraded" to the latest (5.0.4) npm and started having this issue where my npm-linked deps are disappearing after every npm install. What's even weirder is that one of the deps of the linked package is seemingly disappearing as well, even though it's a regular npm package.

So I have to do npm install in the linked package and npm link my-package after every time npm does anything. OHHHHH.

FUN TIMES>>

@mark007
Copy link

@mark007 mark007 commented Jul 4, 2017

I'm also seeing npm prune deleting dependencies of linked modules. I have a feeling this is the cause of alot of other existing npm issues that are still open. It looks like link and prune are not compatible with eachother at the moment.

@shazron
Copy link

@shazron shazron commented Jul 10, 2017

Some info in this Twitter thread between @indutny, @zkat, and @iarna
https://twitter.com/indutny/status/883742222419623937

tunylund pushed a commit to alphagov/passport-verify-stub-relying-party that referenced this issue Jul 17, 2017
https://yarnpkg.com/en/

We've been having a bad time with npm due to this issue:

npm/npm#10343

Let's try switching to yarn and see if that's less painful?

Authors: @richardTowers @georgievh
@ghost
Copy link

@ghost ghost commented Sep 2, 2017

A quick comment on the stealing problem + local development with symlinks and particularly since this has been open for a long time:

  1. Don't use npm3 or npm4.
  2. Skip over npm5 for the moment because it has race conditions which will be a secondary blocking issue and use yarn for now, although it should be very easy to switch back later.
  3. If you do not need to publish your modules publically the following will work for you, else you will need to consider how and when you commit or avoid committing package.json before a publish.
  4. With npm install ../module or yarn link you can use file specifiers and relative symlinks (e.g. within node_modules, module => ../../module) to avoid the stealing issue but also have symlinked local development. And there should be significant caching gains as well because more of the modules are now locally referenced.
  5. Solution order is clone everything (if you haven't or want to try elsewhere first), yarn link each package (dependency sequence should not matter), and then on a second pass re-link package.json modules for each top-level module (since we've seeded the top-levels this will now work)

I did this using the following as a first test:

.bashrc

MODULES="module1 module2 module3 module4"

function yarn-checkout { time echo $MODULES | tr ' ' '\n' | xargs --max-args=1 --max-procs=8 -I{} git clone git@bitbucket.org:$USER/{}.git;}

function yarn-loop { cd $DIR; for d in ls | egrep -v '?lock'; do echo "## $d"; cd $DIR/$d; eval $1 ; cd $DIR; done;}

function yarn-install { yarn-loop "yarn link";}

alias yarn-lnk='for f in cat package.json|grep file:|sed "s/\"//g"|cut -d: -f1; do yarn link $f; done'
function yarn-auto { yarn-loop "yarn-lnk";}

Upgrades:
npm install -g npm@latest
npm install -g yarn

Above:
#1: Checkout in parallel since you would not expect clone errors and probably not need to read any log output. Parallel was ~20s in our case on roughly 40 private modules as an 8-way.
#2 Just a helper function
#3 Does the initial install on each top-level module.
#4 Does the second pass for private/dependent modules which should work because private linking depth should be one level from the perspective of each top-level.

Your package.json files should now look something like:

{
"author": "First Last username@company.co",
"name": "dev-ops",
"description": "none",
"version": "0.0.1",
"repository": {
"type": "git",
"url": " https://bitbucket.org/username/dev-ops.git"
},
"main": "lib/index.js",
"engines": {
"node": ">= 0.10.26"
},
"dependencies": {
"async-class": "^0.4.1",
"aws-sdk": "^2.3.11",
"bluebird": "^3.3.5",
"cluster-utils": "file:../cluster-utils", <- file specifier, relative path
"config-loader": "file:../config-loader", <- file specifier, relative path
"minimist": "^1.2.0",
"moment": "^2.12.0",
"prompt": "1.0.0",
"q": "^1.4.1",
"supervisor": "~0.6.0",
"underscore": "^1.8.3"
}
}

This worked well for us, moved us away from npm3 + the original npm link strategy, away from global symlinks, is now filesystem layout independent, and was also faster and/or simplifying in many cases in terms of dev-ops. I think the npm5 maintainers have thought about this problem, seems the solution is just not well understood.

Hope this helps.

@jflatow
Copy link

@jflatow jflatow commented Oct 10, 2017

I too noticed this change in behavior, and wish npm install / npm update would ignore symlinks in node_modules as before.

@bertho-zero
Copy link

@bertho-zero bertho-zero commented Oct 26, 2017

I went to yarn who corrected this problem today, npm knows problem since version 3! (yarnpkg/yarn#4757)

@artus9033
Copy link

@artus9033 artus9033 commented Dec 29, 2017

+1 for this still present issue 😞

@miensol
Copy link

@miensol miensol commented Mar 2, 2018

We've experienced an error npm ENOENT: no such file or directory, rename ... this issue with local dependencies between modules inside single project.
E.g.
moduleA:

{
  "dependencies": {
    "aws-sdk": "^2.156.0"
  }
}

moduleB:

{
  "dependencies": {
    "jmespath": "0.15.0",
    "shared": "file:../moduleA"
  }
}

During the npm --prefix moduleA install all packages were installed including a dependency of aws-sdk shared called "jmespath": "0.15.0".

Now the npm --prefix moduleB install removed the moduleA/node_modules/aws-sdk/node_modules/jmespath folder and have put under moduleB/node_modules/jmespath. Now the moduleA node_modules is broken when looked at from the moduleA perspective.

There are probably 2 issues here to be precise. npm --prefix moduleB install symlinked moduleA to moduleB/node_modules/moduleA. However the dependencies of moduleA seemed to not be installed. However, at the same time the command tried to reuse and flatten the dependencies of moduleA.

@UD-UD
Copy link

@UD-UD UD-UD commented Apr 12, 2018

Hi,
A bit late for the reply.
I have developed a cli utility that helps you to keep your local links unbroken even after installing dependencies.
You can download it from npm

Also you can find the source code in github

Hope it helps. :)

@rdsedmundo
Copy link

@rdsedmundo rdsedmundo commented Jun 1, 2018

NPM v6 still have this issue. Almost 3 years later. I'm wondering what's big-bug for the NPM team.

@zkat
Copy link
Contributor

@zkat zkat commented Jun 1, 2018

I want to note that one of our big major changes for the upcoming npm@7 is going to be an overhaul of how npm link works, and this will probably be fully and thoroughly addressed in that release. For the time being, if anyone wants to keep this issue specifically alive, feel free to re-post it to our new npm.community forum, where we'll be moving all our issue tracking soon.

In the meantime, and because this thread is already long and impossible to actually consume, I'm going to close+lock this, because I don't intend to look in here again :)

@zkat zkat closed this Jun 1, 2018
@npm npm locked as resolved and limited conversation to collaborators Jun 1, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.