This repository has been archived by the owner. It is now read-only.

`npm install` after `npm link` will "steal" dependencies from linked packages #10343

Closed
jhenninger opened this Issue Nov 11, 2015 · 90 comments

Comments

Projects
None yet
@jhenninger

Create two packages:

jo@t420:/tmp/npmtest$ mkdir A B

A depends on lodash:

jo@t420:/tmp/npmtest$ cd A
jo@t420:/tmp/npmtest/A$ npm init --yes && npm install --save lodash
Wrote to /tmp/npmtest/A/package.json:

{
  "name": "A",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


A@1.0.0 /tmp/npmtest/A
`-- lodash@3.10.1 

npm WARN EPACKAGEJSON A@1.0.0 No description
npm WARN EPACKAGEJSON A@1.0.0 No repository field.

B depends on A which is symlinked for development:

jo@t420:/tmp/npmtest/A$ cd ../B
jo@t420:/tmp/npmtest/B$ npm init --yes                       
Wrote to /tmp/npmtest/B/package.json:

{
  "name": "B",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Create the link:

jo@t420:/tmp/npmtest/B$ npm link ../A
npm WARN EPACKAGEJSON A@1.0.0 No description
npm WARN EPACKAGEJSON A@1.0.0 No repository field.
/home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A -> /tmp/npmtest/A
/tmp/npmtest/B/node_modules/A -> /home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A -> /tmp/npmtest/A

So far so good, A still has lodash in its node_modules:

jo@t420:/tmp/npmtest/B$ tree -d ..
..
├── A
│   └── node_modules
│       └── lodash
│           ├── array
│           ├── chain
│           ├── collection
│           ├── date
│           ├── function
│           ├── internal
│           ├── lang
│           ├── math
│           ├── number
│           ├── object
│           ├── string
│           └── utility
└── B
    └── node_modules
        └── A -> /home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A

18 directories

Now actually save A as a dependency of B in package.json:

jo@t420:/tmp/npmtest/B$ npm install --save ../A
npm WARN update-linked node_modules/A needs updating to 1.0.0 from 1.0.0 but we can't, as it's a symlink
lodash@3.10.1 node_modules/A/node_modules/lodash -> node_modules/lodash
npm WARN EPACKAGEJSON B@1.0.0 No description
npm WARN EPACKAGEJSON B@1.0.0 No repository field.

And all dependencies of A get moved to B's node_modules folder:

jo@t420:/tmp/npmtest/B$ tree -d ..
..
├── A
└── B
    └── node_modules
        ├── A -> /home/jo/code/nvm/versions/node/v4.2.1/lib/node_modules/A
        └── lodash
            ├── array
            ├── chain
            ├── collection
            ├── date
            ├── function
            ├── internal
            ├── lang
            ├── math
            ├── number
            ├── object
            ├── string
            └── utility

17 directories

Running tests etc. in A will now fail due to missing dependencies, unless you run
npm install in A again.

This will also happen when manually adding A as a dependency to B's package.json
and then running npm install.

npm version: 3.3.12

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Nov 11, 2015

Member

So, I think this is caused by two things:

  1. 33aedaf which caused npm to move stuff around when it shouldn't.

AND

  1. A specific bug that allows moves to happen from inside linked modules. (This https://github.com/npm/npm/blob/v3.4.0/lib/install/diff-trees.js#L145-L152 should not generate moves if the source is inside a link. But further, https://github.com/npm/npm/blob/v3.4.0/lib/install/diff-trees.js#L119-L126 probably shouldn't generate removes if the source is inside a link. Possibly these should be like update-link, which is to say, a different action that just warns.)
Member

iarna commented Nov 11, 2015

So, I think this is caused by two things:

  1. 33aedaf which caused npm to move stuff around when it shouldn't.

AND

  1. A specific bug that allows moves to happen from inside linked modules. (This https://github.com/npm/npm/blob/v3.4.0/lib/install/diff-trees.js#L145-L152 should not generate moves if the source is inside a link. But further, https://github.com/npm/npm/blob/v3.4.0/lib/install/diff-trees.js#L119-L126 probably shouldn't generate removes if the source is inside a link. Possibly these should be like update-link, which is to say, a different action that just warns.)

i-e-b added a commit to i-e-b/npm-workspace that referenced this issue Nov 13, 2015

Changes for npm version 3 compatibility
npm v3 has broken a few behaviours that `npm-workspace` relies on.

- Peer dependencies:
  These are no longer installed by default. When `npm-workspace`
  explicitly adds these, they remain functional. Test have been updated to
  reflect new behaviour.

- Linked dependency flattening ( npm/npm#10343
  and npm/npm#10348 and
  npm/npm#9999 )
  npm v3 does not correctly install dependencies that are shared between a
  package and its sym-linked dependency.
```
  A
  +- B (linked)
  |  +- C
  +- C
```
  Should result in both `A` and `B` getting a copy of `C`, however it
  results in:
```
  A
  +- B (linked)
     +- C
```
  To work around this, npm-workspace now adds 'dummy' packages for ones it
  will link, then runs `npm i` then replaces the dummy packages with the
  real symlinks. This results in a complete and correct install.
  Due to a change in the way npm checks versions after npm 3, the dummy
  packages need to be different, so the npm version is read at the start
  of an install.

@i-e-b i-e-b referenced this issue in mariocasciaro/npm-workspace Nov 13, 2015

Merged

Changes for npm version 3 compatibility #23

@baelter

This comment has been minimized.

Show comment
Hide comment
@baelter

baelter Nov 25, 2015

+1 this is burning time = money and sanity. Target release? Need help?

baelter commented Nov 25, 2015

+1 this is burning time = money and sanity. Target release? Need help?

@ElliotChong

This comment has been minimized.

Show comment
Hide comment
@ElliotChong

ElliotChong Dec 9, 2015

Ghetto hacky workaround idea that might help the next developer: Try using the NODE_PATH env variable to treat B as a node_modules path.

NODE_PATH=/path/to/B/node_modules node index.js

https://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

Ghetto hacky workaround idea that might help the next developer: Try using the NODE_PATH env variable to treat B as a node_modules path.

NODE_PATH=/path/to/B/node_modules node index.js

https://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

@timdp

This comment has been minimized.

Show comment
Hide comment
@timdp

timdp Dec 23, 2015

I've had some success working around this issue by rebuilding my node_modules folder with this script (written for Windows). However, it often triggers "refusing to delete: containing path isn't under npm's control" errors. I'm sure that's a different issue underneath. Regardless, npm link is just broken at this point.

timdp commented Dec 23, 2015

I've had some success working around this issue by rebuilding my node_modules folder with this script (written for Windows). However, it often triggers "refusing to delete: containing path isn't under npm's control" errors. I'm sure that's a different issue underneath. Regardless, npm link is just broken at this point.

@whitecolor

This comment has been minimized.

Show comment
Hide comment
@whitecolor

whitecolor Dec 30, 2015

+1 when it is going to be fixed?

+1 when it is going to be fixed?

@tehsenaus

This comment has been minimized.

Show comment
Hide comment
@tehsenaus

tehsenaus Jan 22, 2016

Any update on this?

Any update on this?

@whitecolor

This comment has been minimized.

Show comment
Hide comment
@whitecolor

whitecolor Jan 22, 2016

That actually want holding many from using npm3.

That actually want holding many from using npm3.

@baelter

This comment has been minimized.

Show comment
Hide comment
@baelter

baelter Jan 26, 2016

@whitecolor Maybe not but it's still an undesirable "feature"

baelter commented Jan 26, 2016

@whitecolor Maybe not but it's still an undesirable "feature"

@baelter

This comment has been minimized.

Show comment
Hide comment
@baelter

baelter Feb 10, 2016

I think the whole link command seems a bit over complicated maybe? Correct me if I'm wrong but wouldn't the following do what we want @iarna ?

npm link x should only do:

  1. force move node_modules/x to node_modules/.x
  2. create symlink

npm unlink [x] should only do:

  1. If no argument, do for all symlinks, else do for x:
  2. remove symlink
  3. move node_modules/.x to node_modules/x

Then you should probably carry on with an npm update, but that should maybe not be a part of linking.

Ps. We should also have a npm list links

baelter commented Feb 10, 2016

I think the whole link command seems a bit over complicated maybe? Correct me if I'm wrong but wouldn't the following do what we want @iarna ?

npm link x should only do:

  1. force move node_modules/x to node_modules/.x
  2. create symlink

npm unlink [x] should only do:

  1. If no argument, do for all symlinks, else do for x:
  2. remove symlink
  3. move node_modules/.x to node_modules/x

Then you should probably carry on with an npm update, but that should maybe not be a part of linking.

Ps. We should also have a npm list links

@larsenjh

This comment has been minimized.

Show comment
Hide comment
@larsenjh

larsenjh Feb 18, 2016

+1 please fix this. We have a very complex module structure, and this problem causes many of us to have to re-install all our deps in all our projects over again. Terrible bug.

+1 please fix this. We have a very complex module structure, and this problem causes many of us to have to re-install all our deps in all our projects over again. Terrible bug.

@timdp

This comment has been minimized.

Show comment
Hide comment
@timdp

timdp Feb 18, 2016

@baelter As I understand it, npm link itself is fine, but npm install doesn't treat links any differently (or maybe it does, but badly), which causes inconsistencies.

timdp commented Feb 18, 2016

@baelter As I understand it, npm link itself is fine, but npm install doesn't treat links any differently (or maybe it does, but badly), which causes inconsistencies.

@baelter

This comment has been minimized.

Show comment
Hide comment
@baelter

baelter Feb 18, 2016

@timdp It looks to me like npm link x does a lot more that what I suggested above. npm link is simple enough and I haven't found any issues with that.

baelter commented Feb 18, 2016

@timdp It looks to me like npm link x does a lot more that what I suggested above. npm link is simple enough and I haven't found any issues with that.

@schmod

This comment has been minimized.

Show comment
Hide comment
@schmod

schmod Mar 25, 2016

Contributor

@timdp Agreed, and I'm not sure whether or not this is intentional. If nothing else, it's a significant departure from npm2.x's behavior.

For example, if I have smallApp that depends on bigLib, I might do something like this:

$ cd bigLib
$ npm install
$ npm link

$ cd ../smallApp
$ npm link bigLib
   ~/smallApp/node_modules/bigLib -> ~/npm/lib/node_modules/bigLib -> ~/bigLib
$ npm install 

If smallApp has no other dependencies, I would expect that final npm install command to do nothing, because all of bigLib's dependencies are already installed in bigLib/node_modules. (This what npm2.x would have done)

Instead, running npm install from smallApp installs a duplicate set of bigLib's dependencies into smallApp's node_modules folder. Because bigLib is big and has many dependencies, this could potentially be a very lengthy operation.


Ideally, npm3 should avoid tree-flattening when traversing into linked packages. I'm having a difficult time envisioning a situation where the current behavior is desirable.


One tricky case worth considering:

If bigLib is missing a dependency, should npm install a copy of the missing dep into smallApp?

Or can we enforce a firm contract that npm does not resolve transitive dependencies for linked modules, thus making the linked module completely responsible for providing its dependencies?)

Contributor

schmod commented Mar 25, 2016

@timdp Agreed, and I'm not sure whether or not this is intentional. If nothing else, it's a significant departure from npm2.x's behavior.

For example, if I have smallApp that depends on bigLib, I might do something like this:

$ cd bigLib
$ npm install
$ npm link

$ cd ../smallApp
$ npm link bigLib
   ~/smallApp/node_modules/bigLib -> ~/npm/lib/node_modules/bigLib -> ~/bigLib
$ npm install 

If smallApp has no other dependencies, I would expect that final npm install command to do nothing, because all of bigLib's dependencies are already installed in bigLib/node_modules. (This what npm2.x would have done)

Instead, running npm install from smallApp installs a duplicate set of bigLib's dependencies into smallApp's node_modules folder. Because bigLib is big and has many dependencies, this could potentially be a very lengthy operation.


Ideally, npm3 should avoid tree-flattening when traversing into linked packages. I'm having a difficult time envisioning a situation where the current behavior is desirable.


One tricky case worth considering:

If bigLib is missing a dependency, should npm install a copy of the missing dep into smallApp?

Or can we enforce a firm contract that npm does not resolve transitive dependencies for linked modules, thus making the linked module completely responsible for providing its dependencies?)

@i-e-b

This comment has been minimized.

Show comment
Hide comment
@i-e-b

i-e-b Mar 25, 2016

@schmod

thus making the linked module completely responsible for providing its dependencies?

I would say this is the most sensible option, both closest to npm 2, and it supports the case where a linked dependency may be part of a wider project and is being included in various places for packaging reasons.

i-e-b commented Mar 25, 2016

@schmod

thus making the linked module completely responsible for providing its dependencies?

I would say this is the most sensible option, both closest to npm 2, and it supports the case where a linked dependency may be part of a wider project and is being included in various places for packaging reasons.

@timdp

This comment has been minimized.

Show comment
Hide comment
@timdp

timdp Mar 27, 2016

I'd just always install/keep all the linked package's dependencies inside its node_modules, regardless of whether you ran npm install for the package itself or for one that links to it. That way, at worst, you'll end up with some duplicate dependencies caused by changes to package.json, which should be easy to fix with npm prune. At the same time, all your packages will have all their dependencies available, no matter how they're getting loaded. But I sense I'm missing something, since I don't understand why this continues to be an issue in npm 3.

timdp commented Mar 27, 2016

I'd just always install/keep all the linked package's dependencies inside its node_modules, regardless of whether you ran npm install for the package itself or for one that links to it. That way, at worst, you'll end up with some duplicate dependencies caused by changes to package.json, which should be easy to fix with npm prune. At the same time, all your packages will have all their dependencies available, no matter how they're getting loaded. But I sense I'm missing something, since I don't understand why this continues to be an issue in npm 3.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Apr 9, 2016

endless frustration with this issue, endless.

ghost commented Apr 9, 2016

endless frustration with this issue, endless.

@othiym23 othiym23 modified the milestone: understandable links Apr 13, 2016

asfgit pushed a commit to apache/cordova-lib that referenced this issue May 11, 2016

@rochoa rochoa referenced this issue in CartoDB/node-mapnik-packaging-recipes May 23, 2016

Merged

Script fixes #1

@velezsarain velezsarain referenced this issue in HabitRPG/habitica May 24, 2016

Merged

Vagrantfile and scripts changes for API v3 #7478

schmod pushed a commit to schmod/npm that referenced this issue May 24, 2016

Andrew Schmadel
Trust linked dependencies without a version specifier
Prevents npm install from crossing into the symlinked package, and
trying to reinstall all of its dependencies

Mitigates #10343
@schmod

This comment has been minimized.

Show comment
Hide comment
@schmod

schmod May 26, 2016

Contributor

I have a PR open that appears to address this issue under the following circumstances:

  • Module B has a symlinked copy of Module A in its node_modules folder. (Possibly, but not necessarily created with npm link)
  • Module B's package.json lists Module A as a dependency or devDependency
  • This dependency is listed without a version specifier (ie. it is a FS or Git dependency)

Has anybody been able to reproduce this issue outside of those constraints? As far as I've been able to tell, this bug is limited to non-repo dependencies.

Contributor

schmod commented May 26, 2016

I have a PR open that appears to address this issue under the following circumstances:

  • Module B has a symlinked copy of Module A in its node_modules folder. (Possibly, but not necessarily created with npm link)
  • Module B's package.json lists Module A as a dependency or devDependency
  • This dependency is listed without a version specifier (ie. it is a FS or Git dependency)

Has anybody been able to reproduce this issue outside of those constraints? As far as I've been able to tell, this bug is limited to non-repo dependencies.

@tellnes

This comment has been minimized.

Show comment
Hide comment
@tellnes

tellnes May 28, 2016

@schmod Yes, I'm seeing this issue for dependencies with version ranges. And I've not jet seen a consistent behavior for when it is stealing and when it is not.

The solutions is usually to rerun npm install a few times in the module that is loosing dependencies after I've run npm install in the module that is stealing them. But this fast gets inconvenient when you have a tree of dependecies that is linked together.

tellnes commented May 28, 2016

@schmod Yes, I'm seeing this issue for dependencies with version ranges. And I've not jet seen a consistent behavior for when it is stealing and when it is not.

The solutions is usually to rerun npm install a few times in the module that is loosing dependencies after I've run npm install in the module that is stealing them. But this fast gets inconvenient when you have a tree of dependecies that is linked together.

@schmod

This comment has been minimized.

Show comment
Hide comment
@schmod

schmod May 31, 2016

Contributor

@tellnes Do you have a simple scenario/repo that can reliably reproduce that behavior?

I spent quite some time trying to reproduce this bug with version ranges, and was unable to do so.

Contributor

schmod commented May 31, 2016

@tellnes Do you have a simple scenario/repo that can reliably reproduce that behavior?

I spent quite some time trying to reproduce this bug with version ranges, and was unable to do so.

@tellnes

This comment has been minimized.

Show comment
Hide comment
@tellnes

tellnes May 31, 2016

@schmod I'll see next week if I can reproduce this in a single setup.

Most of the modules are private, hosted with Sinopia. My current setup have multiple levels of symlinks down in the dependency tree.

tellnes commented May 31, 2016

@schmod I'll see next week if I can reproduce this in a single setup.

Most of the modules are private, hosted with Sinopia. My current setup have multiple levels of symlinks down in the dependency tree.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jun 2, 2016

Member

Just FYI, this is my current priority. This does not seem to be resolved by the fix for #10800, which is quite interesting.

I believe the original scenario should have removed the link to A and installed a copy of it instead, as that's what npm install --save ../A means, I think what was intended was npm install --save --link ../A but I see that doesn't work _either. =/

But from what you all are saying I think the problem you all are typically having is more along the lines of what #10800 is describing, and that, that I have a fix for and a (just now) written super minimal reproducer test, so I expect that to land next week.

Member

iarna commented Jun 2, 2016

Just FYI, this is my current priority. This does not seem to be resolved by the fix for #10800, which is quite interesting.

I believe the original scenario should have removed the link to A and installed a copy of it instead, as that's what npm install --save ../A means, I think what was intended was npm install --save --link ../A but I see that doesn't work _either. =/

But from what you all are saying I think the problem you all are typically having is more along the lines of what #10800 is describing, and that, that I have a fix for and a (just now) written super minimal reproducer test, so I expect that to land next week.

@schmod

This comment has been minimized.

Show comment
Hide comment
@schmod

schmod Jun 2, 2016

Contributor

@iarna If you haven't seen it, I have a PR that mitigates this somewhat, although I hadn't considered the impacts of using --save on a linked package with my fix...

Contributor

schmod commented Jun 2, 2016

@iarna If you haven't seen it, I have a PR that mitigates this somewhat, although I hadn't considered the impacts of using --save on a linked package with my fix...

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jun 2, 2016

Member

@schmod: I just took a looksey, I'll followup over there.

Member

iarna commented Jun 2, 2016

@schmod: I just took a looksey, I'll followup over there.

@dkent600

This comment has been minimized.

Show comment
Hide comment
@dkent600

dkent600 Jun 26, 2017

@chevex If is the keyword there.

@chevex If is the keyword there.

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

@dkent600 It sounded like you were saying we shouldn't worry about it getting fixed because npm link isn't useful in general. My mistake.

Contributor

chevex commented Jun 26, 2017

@dkent600 It sounded like you were saying we shouldn't worry about it getting fixed because npm link isn't useful in general. My mistake.

@dkent600

This comment has been minimized.

Show comment
Hide comment
@dkent600

dkent600 Jun 26, 2017

@chevex I would LOVE to have this tool working properly. I really need it. I'm wasting a ton of time trying to figure out what is going on and how to work around it.

dkent600 commented Jun 26, 2017

@chevex I would LOVE to have this tool working properly. I really need it. I'm wasting a ton of time trying to figure out what is going on and how to work around it.

@basicdays

This comment has been minimized.

Show comment
Hide comment
@basicdays

basicdays Jun 26, 2017

For what it's worth, I just tried npm install <folder>, where <folder> is the local package that I actually want to link to. It seems to create the link in the node_modules directory correctly, it doesn't hoist all the packages modules out of the project, and npm install doesn't screw everything up. Dunno, seems npm install pretty much replaces npm link for my needs at the moment. ¯\_(ツ)_/¯

basicdays commented Jun 26, 2017

For what it's worth, I just tried npm install <folder>, where <folder> is the local package that I actually want to link to. It seems to create the link in the node_modules directory correctly, it doesn't hoist all the packages modules out of the project, and npm install doesn't screw everything up. Dunno, seems npm install pretty much replaces npm link for my needs at the moment. ¯\_(ツ)_/¯

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

@basicdays Yes that will put it in your node_modules folder but you need to do that every time you make a change to <folder>. You can't simply make a change and then run your app. You need to reinstall it every time.

Contributor

chevex commented Jun 26, 2017

@basicdays Yes that will put it in your node_modules folder but you need to do that every time you make a change to <folder>. You can't simply make a change and then run your app. You need to reinstall it every time.

@basicdays

This comment has been minimized.

Show comment
Hide comment
@basicdays

basicdays Jun 26, 2017

@chevex That doesn't seem to be what I'm seeing. The actual file that's in the main project's node_modules is indeed the sym link that we want to the linked module. The package.json does reflect that it's a local file dependency. When I run npm install again later, it correctly doesn't clobber the linked package's node_modules. When I make an update to the linked package, the main project seems to automatically get the updated code. Like, this is exactly what npm link should be doing, I have no idea why it's npm install that does it correctly (on my machine?).

@chevex That doesn't seem to be what I'm seeing. The actual file that's in the main project's node_modules is indeed the sym link that we want to the linked module. The package.json does reflect that it's a local file dependency. When I run npm install again later, it correctly doesn't clobber the linked package's node_modules. When I make an update to the linked package, the main project seems to automatically get the updated code. Like, this is exactly what npm link should be doing, I have no idea why it's npm install that does it correctly (on my machine?).

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

Well now I have to try lol. Will report back.

Contributor

chevex commented Jun 26, 2017

Well now I have to try lol. Will report back.

@basicdays

This comment has been minimized.

Show comment
Hide comment
@basicdays

basicdays Jun 26, 2017

Please do! I really hope it works on more than just my machine. :)

Please do! I really hope it works on more than just my machine. :)

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

This does work. In package.json it adds a dependency with the format file:<path-to-file>.
Example:

"dependencies": {
  "my-library": "file:../my-library"
}

npm link doesn't modify package.json at all which would be preferred in order to preserve the real dependency without accidentally committing this hard reference, but the above does appear to work as expected on OS X. It maintains a symlink to the specified file and re-running npm install does not nuke anything in the linked projects' folder. Will try Windows tonight. Thanks for that @basicdays!

Contributor

chevex commented Jun 26, 2017

This does work. In package.json it adds a dependency with the format file:<path-to-file>.
Example:

"dependencies": {
  "my-library": "file:../my-library"
}

npm link doesn't modify package.json at all which would be preferred in order to preserve the real dependency without accidentally committing this hard reference, but the above does appear to work as expected on OS X. It maintains a symlink to the specified file and re-running npm install does not nuke anything in the linked projects' folder. Will try Windows tonight. Thanks for that @basicdays!

@alan-agius4

This comment has been minimized.

Show comment
Hide comment
@alan-agius4

alan-agius4 Jun 26, 2017

alan-agius4 commented Jun 26, 2017

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

@alan-agius4 I tested with the latest version of npm (5.0.3).

Contributor

chevex commented Jun 26, 2017

@alan-agius4 I tested with the latest version of npm (5.0.3).

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

So I think @basicdays may have discovered the fix they implemented here and this issue was simply left open/forgotten.

I just decided to go back and repro the original issue. First I did it with npm version 4.2.0.

$ mkdir A B
$ cd A
$ npm init --yes
$ npm install --save lodash
$ cd ../B
$ npm init --yes
$ npm link ../A
$ npm install --save ../A

npm4-link-issue

This correctly reproduced the issue. Lodash disappeared from A's node_modules and appeared in B's node_modules. Notice that the reproduction steps link to the module and then install the module from the local file system.

I verified that in npm 4 it did indeed copy the module from the referenced path into B's node_modules folder. It did not create a symlink, hence the need for npm link in addition to local file system npm install.

I then went forward and tried the reproduction steps in npm@5 and got some interesting results.

npm5-link-issue

Notice that when I did npm install ../A portion after doing npm link ../A it just nuked A's node_modules folder. I'm not sure why that happened but B's package.json file did get updated with a reference to A on the file system so I decided to just run npm install after that. Suddenly the tree looked normal with lodash living in A's node_modules folder as expected.

What I am taking away from the above is that npm link felt redundant because you can just do this:

npm5-link-issue-solved

One thing I've yet to do is test npm link with the package installed from the registry instead of the file system. I'll do that next. Installing from file system now seems to create a symlink whereas in npm@4 it actually copied the files, making it a better simulation of installing from the npm registry. Since it now creates a symlink the original reproduction steps don't apply. For now it looks like @basicdays has figured out a hell of a workaround (or possible official solution that nobody has updated us about). I'll keep testing around but this behavior hints to me that it's intentional.

Contributor

chevex commented Jun 26, 2017

So I think @basicdays may have discovered the fix they implemented here and this issue was simply left open/forgotten.

I just decided to go back and repro the original issue. First I did it with npm version 4.2.0.

$ mkdir A B
$ cd A
$ npm init --yes
$ npm install --save lodash
$ cd ../B
$ npm init --yes
$ npm link ../A
$ npm install --save ../A

npm4-link-issue

This correctly reproduced the issue. Lodash disappeared from A's node_modules and appeared in B's node_modules. Notice that the reproduction steps link to the module and then install the module from the local file system.

I verified that in npm 4 it did indeed copy the module from the referenced path into B's node_modules folder. It did not create a symlink, hence the need for npm link in addition to local file system npm install.

I then went forward and tried the reproduction steps in npm@5 and got some interesting results.

npm5-link-issue

Notice that when I did npm install ../A portion after doing npm link ../A it just nuked A's node_modules folder. I'm not sure why that happened but B's package.json file did get updated with a reference to A on the file system so I decided to just run npm install after that. Suddenly the tree looked normal with lodash living in A's node_modules folder as expected.

What I am taking away from the above is that npm link felt redundant because you can just do this:

npm5-link-issue-solved

One thing I've yet to do is test npm link with the package installed from the registry instead of the file system. I'll do that next. Installing from file system now seems to create a symlink whereas in npm@4 it actually copied the files, making it a better simulation of installing from the npm registry. Since it now creates a symlink the original reproduction steps don't apply. For now it looks like @basicdays has figured out a hell of a workaround (or possible official solution that nobody has updated us about). I'll keep testing around but this behavior hints to me that it's intentional.

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 26, 2017

Contributor

I can't reproduce the original bug of the linked project's dependencies being flattened into the parent project's dependencies with npm@5, not even with a library that is on the registry. So the original bug isn't an issue anymore. Here I'll go through the scenarios I set up and why I think this new behavior is intentional. Whether or not we users should be happy with it or not is up for discussion.

Here are the following scenarios I've tested with a module that lives on the registry and on my machine somewhere (remember that --save is implied in npm@5+):

Just a quick note for anyone confused about npm link:

cd app && npm link path/to/dependency
is equivalent to
cd path/to/dependency && npm link
followed by
cd app && npm link dependency-name

Both ways create a symlink from the dependency to npm's global node_modules and then a symlink from the app to that global symlink.

Creating link after install.

$ mkdir app && cd app
$ npm init -y
$ npm install registry-package
$ npm link local/path/to/registry-package

In this scenario the installed registry package is overwritten by the symlink. Any installed modules in the local symlinked package seem untouched 🎉

Installing package after linking it.

$ mkdir app && cd app
$ npm init -y
$ npm link local/path/to/registry-package

This so far correctly links to the local package and that package's dependencies are untouched 🎉

But then if you:

$ npm install registry-package

It will overwrite the symlink with the actual package from the registry. However there is a weird side effect here. The previously linked package who's symlink just got overwritten also gets its node_modules directory nuked from existence for seemingly no reason 😠

What I am taking away from this is that npm link no longer seems intended to be used to create a permanent local link. It's more for a quick link for debugging. For example, if you're having problems in your app and you've traced it to a dependency then you can quickly npm link path/to/dependency and you're good to debug real quick without affecting your app's package.json reference to the actual registry package.

If you want a permanent link to a local package, say for an app still in active development, you would run npm install path/to/dependency which permanently symlinks the package into your app with a "dependency": "file:path/to/dependency" reference in your package.json. Then when you're ready to go live you would publish your dependencies and replace those references in your app with the actual registry packages.

This is all poorly documented and nobody has stopped by this issue to update us, but this behavior seems deliberate to me. I think this particular issue can be closed. Though I might want to open a new issue for that odd side effect where installing over an existing symlink gives the symlinked package a courtesy wipe of its node_modules for no reason.

Contributor

chevex commented Jun 26, 2017

I can't reproduce the original bug of the linked project's dependencies being flattened into the parent project's dependencies with npm@5, not even with a library that is on the registry. So the original bug isn't an issue anymore. Here I'll go through the scenarios I set up and why I think this new behavior is intentional. Whether or not we users should be happy with it or not is up for discussion.

Here are the following scenarios I've tested with a module that lives on the registry and on my machine somewhere (remember that --save is implied in npm@5+):

Just a quick note for anyone confused about npm link:

cd app && npm link path/to/dependency
is equivalent to
cd path/to/dependency && npm link
followed by
cd app && npm link dependency-name

Both ways create a symlink from the dependency to npm's global node_modules and then a symlink from the app to that global symlink.

Creating link after install.

$ mkdir app && cd app
$ npm init -y
$ npm install registry-package
$ npm link local/path/to/registry-package

In this scenario the installed registry package is overwritten by the symlink. Any installed modules in the local symlinked package seem untouched 🎉

Installing package after linking it.

$ mkdir app && cd app
$ npm init -y
$ npm link local/path/to/registry-package

This so far correctly links to the local package and that package's dependencies are untouched 🎉

But then if you:

$ npm install registry-package

It will overwrite the symlink with the actual package from the registry. However there is a weird side effect here. The previously linked package who's symlink just got overwritten also gets its node_modules directory nuked from existence for seemingly no reason 😠

What I am taking away from this is that npm link no longer seems intended to be used to create a permanent local link. It's more for a quick link for debugging. For example, if you're having problems in your app and you've traced it to a dependency then you can quickly npm link path/to/dependency and you're good to debug real quick without affecting your app's package.json reference to the actual registry package.

If you want a permanent link to a local package, say for an app still in active development, you would run npm install path/to/dependency which permanently symlinks the package into your app with a "dependency": "file:path/to/dependency" reference in your package.json. Then when you're ready to go live you would publish your dependencies and replace those references in your app with the actual registry packages.

This is all poorly documented and nobody has stopped by this issue to update us, but this behavior seems deliberate to me. I think this particular issue can be closed. Though I might want to open a new issue for that odd side effect where installing over an existing symlink gives the symlinked package a courtesy wipe of its node_modules for no reason.

@dcaillibaud

This comment has been minimized.

Show comment
Hide comment
@dcaillibaud

dcaillibaud Jun 27, 2017

Here is an history of this bug.

  • npm2 : all is fine and work as expected (npm link create symlink and npm install doesn't change anything in linked packages)
  • npm3 & npm4 : npm link create symlink, but npm install move dependencies of the linked package under the main project => linked package himself is broken, you need to re-run npm install in it
  • npm5 : same bug as npm3 & 4, but it's worse as npm install destroy all symlinks

My need is to work with a modular approach, here is a sample hierarchy

  • projectA
    -- myModule1
    --- myModule2
    -- myModule3
    --- myModule2
    -- projectB (can be run as standalone app or embedded app)
    --- myModule4

  • projectC
    -- myModule1
    --- myModule2
    -- myModule4

with npm2, npm link create symlink and I can have all my code at only one place on my disk, so during development changing something in a project dependency change it everywhere, if it broke something I see it immediatly (as long as a test exists and is running in background), without pushing it on a repo increasing version, re-install it on another project changing the project package.json and seeing then something missing.

with npm3 & 4, I need to re-run npm install in all my own dependencies after doing it in a project.

npm5 is clearly unusable in this case.

I don't want to use lib approach with link in the package.json, because I want each module to have it's own repo, and no link in production.

dcaillibaud commented Jun 27, 2017

Here is an history of this bug.

  • npm2 : all is fine and work as expected (npm link create symlink and npm install doesn't change anything in linked packages)
  • npm3 & npm4 : npm link create symlink, but npm install move dependencies of the linked package under the main project => linked package himself is broken, you need to re-run npm install in it
  • npm5 : same bug as npm3 & 4, but it's worse as npm install destroy all symlinks

My need is to work with a modular approach, here is a sample hierarchy

  • projectA
    -- myModule1
    --- myModule2
    -- myModule3
    --- myModule2
    -- projectB (can be run as standalone app or embedded app)
    --- myModule4

  • projectC
    -- myModule1
    --- myModule2
    -- myModule4

with npm2, npm link create symlink and I can have all my code at only one place on my disk, so during development changing something in a project dependency change it everywhere, if it broke something I see it immediatly (as long as a test exists and is running in background), without pushing it on a repo increasing version, re-install it on another project changing the project package.json and seeing then something missing.

with npm3 & 4, I need to re-run npm install in all my own dependencies after doing it in a project.

npm5 is clearly unusable in this case.

I don't want to use lib approach with link in the package.json, because I want each module to have it's own repo, and no link in production.

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 27, 2017

Contributor

I'm not saying it's right or wrong the way it is now, especially since all we have is radio silence. But I'm not sure I understand why each module can't have it's own repo in the setup I demonstrated above? Personally I kind of like that I can npm install ./path/to/myModule and get a permanent symlink because it allows me to nuke my app's node_modules and run npm install and have full confidence my symlinks will return, with their own dependencies intact as well. In npm 4 and lower running npm install ./path/to/myModule did not create a symlink but it does in 5.

My only worry is that I might accidentally commit that reference one day and break my app's build. If that's all you mean then I agree that's a potential nuisance. I'd almost prefer to have a separate file that gets used as an override during npm install and that file could contain the symlink references but be ignored in source control.

I do however see potential with this if I were to say, swap those references out for the real thing in a build script. Then in dev I could have permanent symlinks that I never have to restore, even if I totally delete node_modules in my app. It seems to me that hierarchy you described is definitely doable in npm@5 with the only downside being that your package.json would have "yourModule": "file:./path/to/yourModule" and you'd have to replace those with actual registry references before going to staging/production.

For example, you could potentially create something like a sym-map.json file as a manifest of what to swap out when it's time to go live.

// projectA/package.json

{
  ...
  "dependencies": {
    "myModule1": "file:../myModule1",
    "myModule3": "file:../myModule3",
    "projectB": "file:../projectB"
  }
}
// projectA/sym-map.json

{
  "myModule1": "^3.9.1",
  "myModule3": "^1.2.1",
  "projectB": "^4.0.3"
}

Then just have your build script reference that file during build and swap them out.

var manifest = require('package.json');
var symMap = require('sym-map.json');
var fs = require('fs');

Object.keys(symMap).forEach((name) => {
  manifest.dependencies[name] = symMap[name];
});

fs.writeFile('./package.json', JSON.stringify(manifest));

You could even turn that into a script you could run at any time to swap back and forth.

// swap-references.js

var manifest = require('package.json');
var symMap = require('sym-map.json');
var fs = require('fs');

Object.keys(symMap).forEach((name) => {
  let manifestValue = manifest.dependencies[name];
  manifest.dependencies[name] = symMap[name];
  symMap[name] = manifestValue;
});

fs.writeFile('./package.json', JSON.stringify(manifest));
fs.writeFile('./sym-map.json', JSON.stringify(symMap));

Then run node swap-references.js to switch back and forth at will.

It's a little more work but I think the package.json reference might actually provide more utility by making your symlinks more permanent while in dev. It's more explicit rather than expecting npm install to detect and skip over symlinked modules in node_modules. Not to mention that in the old way if you nuked node_modules in your app then you'd have to do the npm link yourModule all over again.

Anyway, just an idea. Not saying you're wrong at all. I just want to understand your needs better to see if there is indeed no way to accomplish what you're after with npm@5.

Contributor

chevex commented Jun 27, 2017

I'm not saying it's right or wrong the way it is now, especially since all we have is radio silence. But I'm not sure I understand why each module can't have it's own repo in the setup I demonstrated above? Personally I kind of like that I can npm install ./path/to/myModule and get a permanent symlink because it allows me to nuke my app's node_modules and run npm install and have full confidence my symlinks will return, with their own dependencies intact as well. In npm 4 and lower running npm install ./path/to/myModule did not create a symlink but it does in 5.

My only worry is that I might accidentally commit that reference one day and break my app's build. If that's all you mean then I agree that's a potential nuisance. I'd almost prefer to have a separate file that gets used as an override during npm install and that file could contain the symlink references but be ignored in source control.

I do however see potential with this if I were to say, swap those references out for the real thing in a build script. Then in dev I could have permanent symlinks that I never have to restore, even if I totally delete node_modules in my app. It seems to me that hierarchy you described is definitely doable in npm@5 with the only downside being that your package.json would have "yourModule": "file:./path/to/yourModule" and you'd have to replace those with actual registry references before going to staging/production.

For example, you could potentially create something like a sym-map.json file as a manifest of what to swap out when it's time to go live.

// projectA/package.json

{
  ...
  "dependencies": {
    "myModule1": "file:../myModule1",
    "myModule3": "file:../myModule3",
    "projectB": "file:../projectB"
  }
}
// projectA/sym-map.json

{
  "myModule1": "^3.9.1",
  "myModule3": "^1.2.1",
  "projectB": "^4.0.3"
}

Then just have your build script reference that file during build and swap them out.

var manifest = require('package.json');
var symMap = require('sym-map.json');
var fs = require('fs');

Object.keys(symMap).forEach((name) => {
  manifest.dependencies[name] = symMap[name];
});

fs.writeFile('./package.json', JSON.stringify(manifest));

You could even turn that into a script you could run at any time to swap back and forth.

// swap-references.js

var manifest = require('package.json');
var symMap = require('sym-map.json');
var fs = require('fs');

Object.keys(symMap).forEach((name) => {
  let manifestValue = manifest.dependencies[name];
  manifest.dependencies[name] = symMap[name];
  symMap[name] = manifestValue;
});

fs.writeFile('./package.json', JSON.stringify(manifest));
fs.writeFile('./sym-map.json', JSON.stringify(symMap));

Then run node swap-references.js to switch back and forth at will.

It's a little more work but I think the package.json reference might actually provide more utility by making your symlinks more permanent while in dev. It's more explicit rather than expecting npm install to detect and skip over symlinked modules in node_modules. Not to mention that in the old way if you nuked node_modules in your app then you'd have to do the npm link yourModule all over again.

Anyway, just an idea. Not saying you're wrong at all. I just want to understand your needs better to see if there is indeed no way to accomplish what you're after with npm@5.

@laat

This comment has been minimized.

Show comment
Hide comment
@laat

laat Jun 27, 2017

EDIT: Nevermind, I cannot make this work with deeper symlink chains pkg1 -> pkg2 -> pkg3
Ignore this comment.

This seems to work in my tests, as it is not breaking symlinks or steals dependencies

$ npm -v
5.0.4

$ npm link other
$ npm install --preserve-symlinks --no-shrinkwrap

But, as in the original bug report this still steals dependencies:
(the flags do not seem to matter)

$ npm link other
$ npm install --preserve-symlinks --no-shrinkwrap ../other

I would love to see this issue fixed, preserving the symlink behaviour from npm@2 on npm install

laat commented Jun 27, 2017

EDIT: Nevermind, I cannot make this work with deeper symlink chains pkg1 -> pkg2 -> pkg3
Ignore this comment.

This seems to work in my tests, as it is not breaking symlinks or steals dependencies

$ npm -v
5.0.4

$ npm link other
$ npm install --preserve-symlinks --no-shrinkwrap

But, as in the original bug report this still steals dependencies:
(the flags do not seem to matter)

$ npm link other
$ npm install --preserve-symlinks --no-shrinkwrap ../other

I would love to see this issue fixed, preserving the symlink behaviour from npm@2 on npm install

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 27, 2017

Contributor

@laat there's a few issues with your examples.

  1. Those flags aren't causing the difference between your two scenarios (and I'm pretty sure npm has no --preserve-symlinks flag as that is a flag you pass to node, not npm.)

  2. npm link does not modify package.json. So running npm link other and then npm install has no conflicts because npm link didn't add anything to package.json and npm install by itself just reads from package.json to perform installs.

  3. Your second scenario is exactly what I described above with npm 5. If you do npm link other and then npm install ../other the symlink created by npm link other gets overwritten by a symlink created by npm install ../other. Npm 4 and earlier did not create symlinks if you did npm install ../some/path but in npm 5 it does create one, making npm link redundant. The difference is that npm install ../other does modify package.json and adds this explicit reference:

    "dependencies": {
      "other": "file:../other"
    }
    

    With that reference in package.json npm install will now always create a symlink to that file path, even if you delete node_modules altogether and run npm install fresh.

  4. Your second scenario doesn't actually "steal" dependencies from "other". There's an odd side effect when you use npm install ../other to overwrite a symlink created by npm link other where it straight up deletes the node_modules directory from "other".

Basically I believe they've changed how they expect us to use npm link and just haven't updated us here or the documentation.

  • If you want a temporary link to "other" then you run npm link other and overwrite the installed module with a symlink. Running npm install after that will overwrite the symlink with the actual package from the registry again. npm link is not intended to create a permanent link anymore.

  • If you want a permanent link to "other" then you run only npm install ../other without using npm link at all. It adds a reference to package.json and creates a permanent symlink to "other" that won't be overwritten unless you change the reference in package.json to point to the registry. This allows you to run npm install as much as you want and it won't affect that permanent link to "other". To change it to point to an actual package you would run npm install other changing package.json from "other": "file:../other" to the traditional format "other": "^1.2.3". which points to the registry.

Contributor

chevex commented Jun 27, 2017

@laat there's a few issues with your examples.

  1. Those flags aren't causing the difference between your two scenarios (and I'm pretty sure npm has no --preserve-symlinks flag as that is a flag you pass to node, not npm.)

  2. npm link does not modify package.json. So running npm link other and then npm install has no conflicts because npm link didn't add anything to package.json and npm install by itself just reads from package.json to perform installs.

  3. Your second scenario is exactly what I described above with npm 5. If you do npm link other and then npm install ../other the symlink created by npm link other gets overwritten by a symlink created by npm install ../other. Npm 4 and earlier did not create symlinks if you did npm install ../some/path but in npm 5 it does create one, making npm link redundant. The difference is that npm install ../other does modify package.json and adds this explicit reference:

    "dependencies": {
      "other": "file:../other"
    }
    

    With that reference in package.json npm install will now always create a symlink to that file path, even if you delete node_modules altogether and run npm install fresh.

  4. Your second scenario doesn't actually "steal" dependencies from "other". There's an odd side effect when you use npm install ../other to overwrite a symlink created by npm link other where it straight up deletes the node_modules directory from "other".

Basically I believe they've changed how they expect us to use npm link and just haven't updated us here or the documentation.

  • If you want a temporary link to "other" then you run npm link other and overwrite the installed module with a symlink. Running npm install after that will overwrite the symlink with the actual package from the registry again. npm link is not intended to create a permanent link anymore.

  • If you want a permanent link to "other" then you run only npm install ../other without using npm link at all. It adds a reference to package.json and creates a permanent symlink to "other" that won't be overwritten unless you change the reference in package.json to point to the registry. This allows you to run npm install as much as you want and it won't affect that permanent link to "other". To change it to point to an actual package you would run npm install other changing package.json from "other": "file:../other" to the traditional format "other": "^1.2.3". which points to the registry.

@dcaillibaud

This comment has been minimized.

Show comment
Hide comment
@dcaillibaud

dcaillibaud Jun 27, 2017

  1. I want permanent link in my dev env, because my apps and their dependencies are in a living stage, not just today. having to run npm link on every dependency of my own is really painfull, and lead to a lot of duplicates (dependencies of dependencies, placed at the root level by the npm install)
  2. I don't want to put "dependency": "file:path/to/dependency" in my package.json because path/to/dependency is specific on my local machine, others people coding on this app have others path (or none, if they don't touch dependencies code), and in production I don't want any link.
  1. I want permanent link in my dev env, because my apps and their dependencies are in a living stage, not just today. having to run npm link on every dependency of my own is really painfull, and lead to a lot of duplicates (dependencies of dependencies, placed at the root level by the npm install)
  2. I don't want to put "dependency": "file:path/to/dependency" in my package.json because path/to/dependency is specific on my local machine, others people coding on this app have others path (or none, if they don't touch dependencies code), and in production I don't want any link.
@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 27, 2017

Contributor

@dcaillibaud Right, I think I understand. You don't want to check in "dependency": "file:path/to/dependency". That's the one thing I agree is an added nuisance, but there are ways around it like I described above with my little script to swap them out at build/publish time.

What I wish they had done instead is allow us to create some kind of local-dependencies.json file and store those symlink references in there instead of package.json. And have npm install respect that file as an override for dependencies listed in package.json. Then you could just add local-dependencies.json to .gitignore and not have to modify package.json at all. That'd be the best of both worlds. You could totally delete node_modues and run npm install fresh locally and your symlinks would be restored (something you couldn't even do in npm 2 without having to redo all your links) and package.json wouldn't have to have hard references that need changed before being committed or published to production.

Contributor

chevex commented Jun 27, 2017

@dcaillibaud Right, I think I understand. You don't want to check in "dependency": "file:path/to/dependency". That's the one thing I agree is an added nuisance, but there are ways around it like I described above with my little script to swap them out at build/publish time.

What I wish they had done instead is allow us to create some kind of local-dependencies.json file and store those symlink references in there instead of package.json. And have npm install respect that file as an override for dependencies listed in package.json. Then you could just add local-dependencies.json to .gitignore and not have to modify package.json at all. That'd be the best of both worlds. You could totally delete node_modues and run npm install fresh locally and your symlinks would be restored (something you couldn't even do in npm 2 without having to redo all your links) and package.json wouldn't have to have hard references that need changed before being committed or published to production.

@dcaillibaud

This comment has been minimized.

Show comment
Hide comment
@dcaillibaud

dcaillibaud Jun 30, 2017

@chevex It would be great if npm could manage local overrides through e.g. --local-overrides package.local.json

Other solution could be to pass the package.json content to the preinstall script (and be aware of changes), because I tried to modify package.json file at preinstall runtime, but npm doesn't re-read the file before installing.

So, I tried your solution with this script (github don't take js extension, so I put txt), but the syntax `"myModule": "file:../myModule" seems to copy files, not link directory !

This proposition could solve the pb, but in a partial way, still need a script in replacement of npm install, or one of the above suggestion to put this script in preinstall.

@chevex It would be great if npm could manage local overrides through e.g. --local-overrides package.local.json

Other solution could be to pass the package.json content to the preinstall script (and be aware of changes), because I tried to modify package.json file at preinstall runtime, but npm doesn't re-read the file before installing.

So, I tried your solution with this script (github don't take js extension, so I put txt), but the syntax `"myModule": "file:../myModule" seems to copy files, not link directory !

This proposition could solve the pb, but in a partial way, still need a script in replacement of npm install, or one of the above suggestion to put this script in preinstall.

@dcaillibaud

This comment has been minimized.

Show comment
Hide comment
@dcaillibaud

dcaillibaud Jun 30, 2017

I made a script which replace npm install

  • merge package.local-extras.json into package.json, stripping all "file:" dependencies (I don't want a copy)
  • run npm install
  • link all local "file:" dependencies
  • restore original package.json

Use it at your own risk !
https://github.com/Sesamath/sesatheque/blob/0.9.74/scripts/npmi.js

I made a script which replace npm install

  • merge package.local-extras.json into package.json, stripping all "file:" dependencies (I don't want a copy)
  • run npm install
  • link all local "file:" dependencies
  • restore original package.json

Use it at your own risk !
https://github.com/Sesamath/sesatheque/blob/0.9.74/scripts/npmi.js

@chevex

This comment has been minimized.

Show comment
Hide comment
@chevex

chevex Jun 30, 2017

Contributor

You say it copied the files? You sure you were using npm 5? For me that file prefix always creates a Symlink to the directory.

Contributor

chevex commented Jun 30, 2017

You say it copied the files? You sure you were using npm 5? For me that file prefix always creates a Symlink to the directory.

@dcaillibaud

This comment has been minimized.

Show comment
Hide comment
@dcaillibaud

dcaillibaud Jun 30, 2017

I saw this with npm@5.0.3, I'm pretty sure to have seen plain directories after a rm -rf node_modules; npm i (with some file: dependencies), but I can't reproduce it.

I just upgraded to 5.0.4, and I don't really understand, after erasing ~/.npm, putting my file:… in package.json, npm install doesn't install any of the file:… dependencies. If y specify explicitly one of my own module with npm install myPackage, the symlink is created. Then a rm -rf node_modules; npm i install registry modules and myPackage is linked, but other packages with file: aren't ! Like if you need to first install once the local module alone to force npm realize that the path is a real one…
It's the same after a revert to 5.0.3

I saw this with npm@5.0.3, I'm pretty sure to have seen plain directories after a rm -rf node_modules; npm i (with some file: dependencies), but I can't reproduce it.

I just upgraded to 5.0.4, and I don't really understand, after erasing ~/.npm, putting my file:… in package.json, npm install doesn't install any of the file:… dependencies. If y specify explicitly one of my own module with npm install myPackage, the symlink is created. Then a rm -rf node_modules; npm i install registry modules and myPackage is linked, but other packages with file: aren't ! Like if you need to first install once the local module alone to force npm realize that the path is a real one…
It's the same after a revert to 5.0.3

@Madd0g

This comment has been minimized.

Show comment
Hide comment
@Madd0g

Madd0g Jul 2, 2017

Just "upgraded" to the latest (5.0.4) npm and started having this issue where my npm-linked deps are disappearing after every npm install. What's even weirder is that one of the deps of the linked package is seemingly disappearing as well, even though it's a regular npm package.

So I have to do npm install in the linked package and npm link my-package after every time npm does anything. OHHHHH.

FUN TIMES>>

Madd0g commented Jul 2, 2017

Just "upgraded" to the latest (5.0.4) npm and started having this issue where my npm-linked deps are disappearing after every npm install. What's even weirder is that one of the deps of the linked package is seemingly disappearing as well, even though it's a regular npm package.

So I have to do npm install in the linked package and npm link my-package after every time npm does anything. OHHHHH.

FUN TIMES>>

@mark007

This comment has been minimized.

Show comment
Hide comment
@mark007

mark007 Jul 4, 2017

I'm also seeing npm prune deleting dependencies of linked modules. I have a feeling this is the cause of alot of other existing npm issues that are still open. It looks like link and prune are not compatible with eachother at the moment.

mark007 commented Jul 4, 2017

I'm also seeing npm prune deleting dependencies of linked modules. I have a feeling this is the cause of alot of other existing npm issues that are still open. It looks like link and prune are not compatible with eachother at the moment.

@shazron

This comment has been minimized.

Show comment
Hide comment

shazron commented Jul 10, 2017

Some info in this Twitter thread between @indutny, @zkat, and @iarna
https://twitter.com/indutny/status/883742222419623937

tunylund pushed a commit to alphagov/passport-verify-stub-relying-party that referenced this issue Jul 17, 2017

bau: Switch from npm to yarn
https://yarnpkg.com/en/

We've been having a bad time with npm due to this issue:

npm/npm#10343

Let's try switching to yarn and see if that's less painful?

Authors: @richardTowers @georgievh
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 2, 2017

A quick comment on the stealing problem + local development with symlinks and particularly since this has been open for a long time:

  1. Don't use npm3 or npm4.
  2. Skip over npm5 for the moment because it has race conditions which will be a secondary blocking issue and use yarn for now, although it should be very easy to switch back later.
  3. If you do not need to publish your modules publically the following will work for you, else you will need to consider how and when you commit or avoid committing package.json before a publish.
  4. With npm install ../module or yarn link you can use file specifiers and relative symlinks (e.g. within node_modules, module => ../../module) to avoid the stealing issue but also have symlinked local development. And there should be significant caching gains as well because more of the modules are now locally referenced.
  5. Solution order is clone everything (if you haven't or want to try elsewhere first), yarn link each package (dependency sequence should not matter), and then on a second pass re-link package.json modules for each top-level module (since we've seeded the top-levels this will now work)

I did this using the following as a first test:

.bashrc

MODULES="module1 module2 module3 module4"

function yarn-checkout { time echo $MODULES | tr ' ' '\n' | xargs --max-args=1 --max-procs=8 -I{} git clone git@bitbucket.org:$USER/{}.git;}

function yarn-loop { cd $DIR; for d in ls | egrep -v '?lock'; do echo "## $d"; cd $DIR/$d; eval $1 ; cd $DIR; done;}

function yarn-install { yarn-loop "yarn link";}

alias yarn-lnk='for f in cat package.json|grep file:|sed "s/\"//g"|cut -d: -f1; do yarn link $f; done'
function yarn-auto { yarn-loop "yarn-lnk";}

Upgrades:
npm install -g npm@latest
npm install -g yarn

Above:
#1: Checkout in parallel since you would not expect clone errors and probably not need to read any log output. Parallel was ~20s in our case on roughly 40 private modules as an 8-way.
#2 Just a helper function
#3 Does the initial install on each top-level module.
#4 Does the second pass for private/dependent modules which should work because private linking depth should be one level from the perspective of each top-level.

Your package.json files should now look something like:

{
"author": "First Last username@company.co",
"name": "dev-ops",
"description": "none",
"version": "0.0.1",
"repository": {
"type": "git",
"url": " https://bitbucket.org/username/dev-ops.git"
},
"main": "lib/index.js",
"engines": {
"node": ">= 0.10.26"
},
"dependencies": {
"async-class": "^0.4.1",
"aws-sdk": "^2.3.11",
"bluebird": "^3.3.5",
"cluster-utils": "file:../cluster-utils", <- file specifier, relative path
"config-loader": "file:../config-loader", <- file specifier, relative path
"minimist": "^1.2.0",
"moment": "^2.12.0",
"prompt": "1.0.0",
"q": "^1.4.1",
"supervisor": "~0.6.0",
"underscore": "^1.8.3"
}
}

This worked well for us, moved us away from npm3 + the original npm link strategy, away from global symlinks, is now filesystem layout independent, and was also faster and/or simplifying in many cases in terms of dev-ops. I think the npm5 maintainers have thought about this problem, seems the solution is just not well understood.

Hope this helps.

ghost commented Sep 2, 2017

A quick comment on the stealing problem + local development with symlinks and particularly since this has been open for a long time:

  1. Don't use npm3 or npm4.
  2. Skip over npm5 for the moment because it has race conditions which will be a secondary blocking issue and use yarn for now, although it should be very easy to switch back later.
  3. If you do not need to publish your modules publically the following will work for you, else you will need to consider how and when you commit or avoid committing package.json before a publish.
  4. With npm install ../module or yarn link you can use file specifiers and relative symlinks (e.g. within node_modules, module => ../../module) to avoid the stealing issue but also have symlinked local development. And there should be significant caching gains as well because more of the modules are now locally referenced.
  5. Solution order is clone everything (if you haven't or want to try elsewhere first), yarn link each package (dependency sequence should not matter), and then on a second pass re-link package.json modules for each top-level module (since we've seeded the top-levels this will now work)

I did this using the following as a first test:

.bashrc

MODULES="module1 module2 module3 module4"

function yarn-checkout { time echo $MODULES | tr ' ' '\n' | xargs --max-args=1 --max-procs=8 -I{} git clone git@bitbucket.org:$USER/{}.git;}

function yarn-loop { cd $DIR; for d in ls | egrep -v '?lock'; do echo "## $d"; cd $DIR/$d; eval $1 ; cd $DIR; done;}

function yarn-install { yarn-loop "yarn link";}

alias yarn-lnk='for f in cat package.json|grep file:|sed "s/\"//g"|cut -d: -f1; do yarn link $f; done'
function yarn-auto { yarn-loop "yarn-lnk";}

Upgrades:
npm install -g npm@latest
npm install -g yarn

Above:
#1: Checkout in parallel since you would not expect clone errors and probably not need to read any log output. Parallel was ~20s in our case on roughly 40 private modules as an 8-way.
#2 Just a helper function
#3 Does the initial install on each top-level module.
#4 Does the second pass for private/dependent modules which should work because private linking depth should be one level from the perspective of each top-level.

Your package.json files should now look something like:

{
"author": "First Last username@company.co",
"name": "dev-ops",
"description": "none",
"version": "0.0.1",
"repository": {
"type": "git",
"url": " https://bitbucket.org/username/dev-ops.git"
},
"main": "lib/index.js",
"engines": {
"node": ">= 0.10.26"
},
"dependencies": {
"async-class": "^0.4.1",
"aws-sdk": "^2.3.11",
"bluebird": "^3.3.5",
"cluster-utils": "file:../cluster-utils", <- file specifier, relative path
"config-loader": "file:../config-loader", <- file specifier, relative path
"minimist": "^1.2.0",
"moment": "^2.12.0",
"prompt": "1.0.0",
"q": "^1.4.1",
"supervisor": "~0.6.0",
"underscore": "^1.8.3"
}
}

This worked well for us, moved us away from npm3 + the original npm link strategy, away from global symlinks, is now filesystem layout independent, and was also faster and/or simplifying in many cases in terms of dev-ops. I think the npm5 maintainers have thought about this problem, seems the solution is just not well understood.

Hope this helps.

@jflatow

This comment has been minimized.

Show comment
Hide comment
@jflatow

jflatow Oct 10, 2017

I too noticed this change in behavior, and wish npm install / npm update would ignore symlinks in node_modules as before.

jflatow commented Oct 10, 2017

I too noticed this change in behavior, and wish npm install / npm update would ignore symlinks in node_modules as before.

@ryanluker ryanluker referenced this issue in peterharding/minimal_local_modules_ts2300 Oct 25, 2017

Closed

apply lerna, fix problems #1

@bertho-zero

This comment has been minimized.

Show comment
Hide comment
@bertho-zero

bertho-zero Oct 26, 2017

I went to yarn who corrected this problem today, npm knows problem since version 3! (yarnpkg/yarn#4757)

I went to yarn who corrected this problem today, npm knows problem since version 3! (yarnpkg/yarn#4757)

@artus9033

This comment has been minimized.

Show comment
Hide comment
@artus9033

artus9033 Dec 29, 2017

+1 for this still present issue 😞

+1 for this still present issue 😞

@miensol

This comment has been minimized.

Show comment
Hide comment
@miensol

miensol Mar 2, 2018

We've experienced an error npm ENOENT: no such file or directory, rename ... this issue with local dependencies between modules inside single project.
E.g.
moduleA:

{
  "dependencies": {
    "aws-sdk": "^2.156.0"
  }
}

moduleB:

{
  "dependencies": {
    "jmespath": "0.15.0",
    "shared": "file:../moduleA"
  }
}

During the npm --prefix moduleA install all packages were installed including a dependency of aws-sdk shared called "jmespath": "0.15.0".

Now the npm --prefix moduleB install removed the moduleA/node_modules/aws-sdk/node_modules/jmespath folder and have put under moduleB/node_modules/jmespath. Now the moduleA node_modules is broken when looked at from the moduleA perspective.

There are probably 2 issues here to be precise. npm --prefix moduleB install symlinked moduleA to moduleB/node_modules/moduleA. However the dependencies of moduleA seemed to not be installed. However, at the same time the command tried to reuse and flatten the dependencies of moduleA.

miensol commented Mar 2, 2018

We've experienced an error npm ENOENT: no such file or directory, rename ... this issue with local dependencies between modules inside single project.
E.g.
moduleA:

{
  "dependencies": {
    "aws-sdk": "^2.156.0"
  }
}

moduleB:

{
  "dependencies": {
    "jmespath": "0.15.0",
    "shared": "file:../moduleA"
  }
}

During the npm --prefix moduleA install all packages were installed including a dependency of aws-sdk shared called "jmespath": "0.15.0".

Now the npm --prefix moduleB install removed the moduleA/node_modules/aws-sdk/node_modules/jmespath folder and have put under moduleB/node_modules/jmespath. Now the moduleA node_modules is broken when looked at from the moduleA perspective.

There are probably 2 issues here to be precise. npm --prefix moduleB install symlinked moduleA to moduleB/node_modules/moduleA. However the dependencies of moduleA seemed to not be installed. However, at the same time the command tried to reuse and flatten the dependencies of moduleA.

@UD-UD

This comment has been minimized.

Show comment
Hide comment
@UD-UD

UD-UD Apr 12, 2018

Hi,
A bit late for the reply.
I have developed a cli utility that helps you to keep your local links unbroken even after installing dependencies.
You can download it from npm

Also you can find the source code in github

Hope it helps. :)

UD-UD commented Apr 12, 2018

Hi,
A bit late for the reply.
I have developed a cli utility that helps you to keep your local links unbroken even after installing dependencies.
You can download it from npm

Also you can find the source code in github

Hope it helps. :)

@mansona mansona referenced this issue in tomdale/ember-cli-addon-tests May 14, 2018

Open

Npm (5) install will override the symlinked addon under test #176

@rdsedmundo

This comment has been minimized.

Show comment
Hide comment
@rdsedmundo

rdsedmundo Jun 1, 2018

NPM v6 still have this issue. Almost 3 years later. I'm wondering what's big-bug for the NPM team.

rdsedmundo commented Jun 1, 2018

NPM v6 still have this issue. Almost 3 years later. I'm wondering what's big-bug for the NPM team.

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Jun 1, 2018

Member

I want to note that one of our big major changes for the upcoming npm@7 is going to be an overhaul of how npm link works, and this will probably be fully and thoroughly addressed in that release. For the time being, if anyone wants to keep this issue specifically alive, feel free to re-post it to our new npm.community forum, where we'll be moving all our issue tracking soon.

In the meantime, and because this thread is already long and impossible to actually consume, I'm going to close+lock this, because I don't intend to look in here again :)

Member

zkat commented Jun 1, 2018

I want to note that one of our big major changes for the upcoming npm@7 is going to be an overhaul of how npm link works, and this will probably be fully and thoroughly addressed in that release. For the time being, if anyone wants to keep this issue specifically alive, feel free to re-post it to our new npm.community forum, where we'll be moving all our issue tracking soon.

In the meantime, and because this thread is already long and impossible to actually consume, I'm going to close+lock this, because I don't intend to look in here again :)

@zkat zkat closed this Jun 1, 2018

@npm npm locked as resolved and limited conversation to collaborators Jun 1, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.