This repository has been archived by the owner. It is now read-only.

Update of local dependencies does not work #7426

Closed
akloeber opened this Issue Feb 22, 2015 · 44 comments

Comments

Projects
None yet
@akloeber

I have some local dependencies on sub-modules in the same git repo like this:

"dependencies": {
  "sub-mod-0": "./sub-mod-0",
  "sub-mod-1": "./sub-mod-1"
}

With this I can do require('sub-mod-0') without having to use something like require('../../sub-mod-0')

Installation via npm install works fine and populates node_modules/sub-mod-0 and node_modules/sub-mod-1 but updating them is not possible without manually deleting node_modules/sub-mod-* beforehand. Neither npm install nor npm update nor npm update sub-module-* have an effect even if the version of the sub-module is incremented.

Environment:
npm v2.6.0
node v0.10.25
Ubuntu 14.04.1 LTS

@smikes

This comment has been minimized.

Show comment
Hide comment
@smikes

smikes Feb 23, 2015

Contributor

I think you may want to use npm link, if what you're hoping to see is live updates of your dependencies in your parent module.

When you depend on a directory via npm install ../whatever, npm actually builds a tarball based on the state at the time of npm install, stores that tarball in the cache, and then installs that tarball from cache, so (as you note) you are not seeing the live dependencies.

With npm link, npm creates a link from its global install location (see prefix) that points to the dependency; then when you link the dependency into your package, npm makes a link from node_modules that points at the target.

In general this is recommended for development, not production though. Can you tell me more about your situation and what you're trying to accomplish?

See https://docs.npmjs.com/cli/link
and https://docs.npmjs.com/files/folders

Contributor

smikes commented Feb 23, 2015

I think you may want to use npm link, if what you're hoping to see is live updates of your dependencies in your parent module.

When you depend on a directory via npm install ../whatever, npm actually builds a tarball based on the state at the time of npm install, stores that tarball in the cache, and then installs that tarball from cache, so (as you note) you are not seeing the live dependencies.

With npm link, npm creates a link from its global install location (see prefix) that points to the dependency; then when you link the dependency into your package, npm makes a link from node_modules that points at the target.

In general this is recommended for development, not production though. Can you tell me more about your situation and what you're trying to accomplish?

See https://docs.npmjs.com/cli/link
and https://docs.npmjs.com/files/folders

@akloeber

This comment has been minimized.

Show comment
Hide comment
@akloeber

akloeber Feb 23, 2015

Well, let me clarify our situation:
We have a larger system that consists of several "components" that use each other. First we had linked them together directly through file system traversal (i.e. require('../../some-module')) which was not very stable across refactorings because when moving things around the paths had to be adjusted.

So we tried to store each sub-module in its own repository and require() them in the parent module as normal. But it appeared that there were often changes/features across sub module (i.e. component) boundaries and having them in different repos was impractible as these had to be split over several commits in different repositories.

So we aggregated them into a single repository again and used local dependencies once they were introduced. That resolved our problems except the issue during deployment described above (currently we work around that by deleting node_modules followed by npm install which is very time-consuming).

We already know about npm link and use it during development. We do not want to use in production as it is not recommended and we want to avoid the extra scripting for the npm link steps.

What would be the right approach in this case?

Well, let me clarify our situation:
We have a larger system that consists of several "components" that use each other. First we had linked them together directly through file system traversal (i.e. require('../../some-module')) which was not very stable across refactorings because when moving things around the paths had to be adjusted.

So we tried to store each sub-module in its own repository and require() them in the parent module as normal. But it appeared that there were often changes/features across sub module (i.e. component) boundaries and having them in different repos was impractible as these had to be split over several commits in different repositories.

So we aggregated them into a single repository again and used local dependencies once they were introduced. That resolved our problems except the issue during deployment described above (currently we work around that by deleting node_modules followed by npm install which is very time-consuming).

We already know about npm link and use it during development. We do not want to use in production as it is not recommended and we want to avoid the extra scripting for the npm link steps.

What would be the right approach in this case?

@smikes

This comment has been minimized.

Show comment
Hide comment
@smikes

smikes Feb 23, 2015

Contributor

What about setting up your NODE_PATH in production so that it searches all of your submodule roots?

http://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

Another option would be to write a wrapper around require but I think you can get what you want just by using NODE_PATH...

Contributor

smikes commented Feb 23, 2015

What about setting up your NODE_PATH in production so that it searches all of your submodule roots?

http://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

Another option would be to write a wrapper around require but I think you can get what you want just by using NODE_PATH...

@akloeber

This comment has been minimized.

Show comment
Hide comment
@akloeber

akloeber Feb 23, 2015

Using NODE_PATH would work but then we still have to do the scripting for executing npm install in every sub-module directory and the content of NODE_PATH has to be managed to keep it in sync with the sub-modules. With the current approach a single execution of npm install populates everything.

We already discussed writing a wrapper/replacement for require but that seems to be a hack and we are still looking for a clean solution.

Using NODE_PATH would work but then we still have to do the scripting for executing npm install in every sub-module directory and the content of NODE_PATH has to be managed to keep it in sync with the sub-modules. With the current approach a single execution of npm install populates everything.

We already discussed writing a wrapper/replacement for require but that seems to be a hack and we are still looking for a clean solution.

@smikes

This comment has been minimized.

Show comment
Hide comment
@smikes

smikes Feb 23, 2015

Contributor

This is getting well beyond my expertise unfortunately -- it is essentially a devops question now. One thing that might work for deployment would be to build the entire package tree once, offline, and pack it up with "npm pack" (possibly using bundledDependencies)

Then you would have a single tgz representing your new version and could deploy it across however many instances you are running.

If you have to support upgrading in-place, then "rm -rf node_modules; npm install" is the best way to be sure... How long is "very time-consuming"?

And of course npm@3 should help with this, because its approach is to build the complete desired dependency graph, calculate the difference between desired and current, and then apply the changes needed to make the transition. Should be cheaper than what you see now.

Contributor

smikes commented Feb 23, 2015

This is getting well beyond my expertise unfortunately -- it is essentially a devops question now. One thing that might work for deployment would be to build the entire package tree once, offline, and pack it up with "npm pack" (possibly using bundledDependencies)

Then you would have a single tgz representing your new version and could deploy it across however many instances you are running.

If you have to support upgrading in-place, then "rm -rf node_modules; npm install" is the best way to be sure... How long is "very time-consuming"?

And of course npm@3 should help with this, because its approach is to build the complete desired dependency graph, calculate the difference between desired and current, and then apply the changes needed to make the transition. Should be cheaper than what you see now.

@akloeber

This comment has been minimized.

Show comment
Hide comment
@akloeber

akloeber Feb 24, 2015

Yes, I suppose uploading a tgz and extracting it might save some time in comparison to rm -rf node_modules; npm install but therefore adds a new build step. When I say "very time-consuming" I mean several minutes which is still OK for production deployment but suboptimal for our CI environment.

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I was hoping for something like npm update --force. Another approach might be optional dependency tracking options regarded by npm update to support fine-tuning for these use cases like:

"dependencies": {
  "lodash": "^2.4.1",
  "sub-mod-0": {
    "path": "./sub-mod-0",
    "tracking": "ALWAYS_SYNC"
  },
  "sub-mod-1": "./sub-mod-1"
}

Nevertheless: Thank you very much for your support and your detailed explanations!

Yes, I suppose uploading a tgz and extracting it might save some time in comparison to rm -rf node_modules; npm install but therefore adds a new build step. When I say "very time-consuming" I mean several minutes which is still OK for production deployment but suboptimal for our CI environment.

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I was hoping for something like npm update --force. Another approach might be optional dependency tracking options regarded by npm update to support fine-tuning for these use cases like:

"dependencies": {
  "lodash": "^2.4.1",
  "sub-mod-0": {
    "path": "./sub-mod-0",
    "tracking": "ALWAYS_SYNC"
  },
  "sub-mod-1": "./sub-mod-1"
}

Nevertheless: Thank you very much for your support and your detailed explanations!

@smikes

This comment has been minimized.

Show comment
Hide comment
@smikes

smikes Feb 24, 2015

Contributor

Here's the blog post that lays out the roadmap for npm@3: http://blog.npmjs.org/post/98233700815/multi-stage-installs-and-a-better-npm

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I think that's not quite right; it's even worse than that. If you are directly installing from git submodules then npm treats them as pretty opaque and does not look into them to do version management. So npm update won't update a git dependency, and you'll have to rely on npm install. It doesn't matter if the commit-ish part of the dependency changes, or if the version in package.json changes.

You may be able to mitigate this a bit by running a private npm registry where you publish your own scoped and versioned modules. Then you can use npm install of prepackaged tarballs in your CI, which should be somewhat faster.

Also you might want to check out npm's paid product, npm Enterprise – I'm not an employee of npm, inc., and I don't actually know precisely what it does. But it may help addressing some of these issues.

Contributor

smikes commented Feb 24, 2015

Here's the blog post that lays out the roadmap for npm@3: http://blog.npmjs.org/post/98233700815/multi-stage-installs-and-a-better-npm

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I think that's not quite right; it's even worse than that. If you are directly installing from git submodules then npm treats them as pretty opaque and does not look into them to do version management. So npm update won't update a git dependency, and you'll have to rely on npm install. It doesn't matter if the commit-ish part of the dependency changes, or if the version in package.json changes.

You may be able to mitigate this a bit by running a private npm registry where you publish your own scoped and versioned modules. Then you can use npm install of prepackaged tarballs in your CI, which should be somewhat faster.

Also you might want to check out npm's paid product, npm Enterprise – I'm not an employee of npm, inc., and I don't actually know precisely what it does. But it may help addressing some of these issues.

@akloeber

This comment has been minimized.

Show comment
Hide comment
@akloeber

akloeber Feb 25, 2015

As far as I understand, at least npm install "updates" git dependencies to the current head. Is that right? (see #1727 and #4104)

I looked at the feature list of npm Enterprise but I could not find anything that might help. A private registry with scoped modules would be similar to our first approach when we used different repos for each submodule. If my assumption on the top is correct we can already achieve by using git URLs to dedicated repos that each contain a submodule. But as I described our modules are not that independent.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install). If it is another invocation of npm install then it should update packages from the registry based on their version, from git based on the git meta data and sync local resources as there is not meta data.

As far as I understand, at least npm install "updates" git dependencies to the current head. Is that right? (see #1727 and #4104)

I looked at the feature list of npm Enterprise but I could not find anything that might help. A private registry with scoped modules would be similar to our first approach when we used different repos for each submodule. If my assumption on the top is correct we can already achieve by using git URLs to dedicated repos that each contain a submodule. But as I described our modules are not that independent.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install). If it is another invocation of npm install then it should update packages from the registry based on their version, from git based on the git meta data and sync local resources as there is not meta data.

@akloeber

This comment has been minimized.

Show comment
Hide comment
@akloeber

akloeber Feb 25, 2015

I found a longer discussion on #1558 with similar use cases and the local paths feature seems to be the "official" fix for those scenarios. If npm install does not update dependencies with local paths then some of them would not be fixed neither.

There is also the suggestion of using preinstall: "npm install sub-module-*" or preinstall: "npm link sub-module-* --prefix=node_modules" in package.json but they are still a kind of workaround.

I found a longer discussion on #1558 with similar use cases and the local paths feature seems to be the "official" fix for those scenarios. If npm install does not update dependencies with local paths then some of them would not be fixed neither.

There is also the suggestion of using preinstall: "npm install sub-module-*" or preinstall: "npm link sub-module-* --prefix=node_modules" in package.json but they are still a kind of workaround.

@smikes

This comment has been minimized.

Show comment
Hide comment
@smikes

smikes Feb 25, 2015

Contributor

wrt npm-enterprise, as I said, I don't know much about the product.

I was thinking it may be faster/ more efficient if your sub-dependencies were already packaged as modules. An npm install from a git repository consists of two clones, a pack, and an install, while npm install from an npm repository is just a download + install, so installing from a prebuilt tarball should be faster.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install).

I don't believe there is a preferred way in npm@2; or rather, "rm -rf node_modules && npm install" is the most succinct way to ensure that this happens

The multi-stage-install core feature of npm@3 has been in development for several months, in order to address the problem of taking the difference between an installation and a specification, and implementing the steps to correct the difference.

Contributor

smikes commented Feb 25, 2015

wrt npm-enterprise, as I said, I don't know much about the product.

I was thinking it may be faster/ more efficient if your sub-dependencies were already packaged as modules. An npm install from a git repository consists of two clones, a pack, and an install, while npm install from an npm repository is just a download + install, so installing from a prebuilt tarball should be faster.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install).

I don't believe there is a preferred way in npm@2; or rather, "rm -rf node_modules && npm install" is the most succinct way to ensure that this happens

The multi-stage-install core feature of npm@3 has been in development for several months, in order to address the problem of taking the difference between an installation and a specification, and implementing the steps to correct the difference.

@smikes

This comment has been minimized.

Show comment
Hide comment
@smikes

smikes Feb 25, 2015

Contributor

You may want to look at #6251 (unfulfilled feature request)

On Wed, Feb 25, 2015 at 10:04 AM, Andreas Klöber notifications@github.com
wrote:

I found a longer discussion on #1558
#1558 with similar use cases and the
local paths feature seems to be the "official" fix for those scenarios. If npm
install does not update dependencies with local paths then some of them
would not be fixed neither.

There is also the suggestion of using preinstall: "npm install
sub-module-" or preinstall: "npm link sub-module- --prefix=node_modules"
in package.json but they are still a kind of workaround.


Reply to this email directly or view it on GitHub
#7426 (comment).

Contributor

smikes commented Feb 25, 2015

You may want to look at #6251 (unfulfilled feature request)

On Wed, Feb 25, 2015 at 10:04 AM, Andreas Klöber notifications@github.com
wrote:

I found a longer discussion on #1558
#1558 with similar use cases and the
local paths feature seems to be the "official" fix for those scenarios. If npm
install does not update dependencies with local paths then some of them
would not be fixed neither.

There is also the suggestion of using preinstall: "npm install
sub-module-" or preinstall: "npm link sub-module- --prefix=node_modules"
in package.json but they are still a kind of workaround.


Reply to this email directly or view it on GitHub
#7426 (comment).

@othiym23 othiym23 added this to the multi-stage install milestone Feb 26, 2015

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Feb 26, 2015

Contributor

I believe this scenario is one of many that will be handled much better by npm@3, which is much more aggressive about making sure that dependencies are brought up to date each time npm install is run, but I don't know how much attention @iarna has given to local dependencies on her work thus far. If she has time, she might drop a note here saying where that's at, or I might take a look at it myself, as the person most responsible for local dependencies in their current state.

Contributor

othiym23 commented Feb 26, 2015

I believe this scenario is one of many that will be handled much better by npm@3, which is much more aggressive about making sure that dependencies are brought up to date each time npm install is run, but I don't know how much attention @iarna has given to local dependencies on her work thus far. If she has time, she might drop a note here saying where that's at, or I might take a look at it myself, as the person most responsible for local dependencies in their current state.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Mar 26, 2015

Member

So with npm@2 I see it updating the local modules. With npm@3 it does not, as the spec from the dependencies in the package.json is met by the version already installed.

However, this does seem like a legitimate bug in npm outdated and thus npm update.

Member

iarna commented Mar 26, 2015

So with npm@2 I see it updating the local modules. With npm@3 it does not, as the spec from the dependencies in the package.json is met by the version already installed.

However, this does seem like a legitimate bug in npm outdated and thus npm update.

@iarna iarna removed this from the multi-stage install milestone Mar 26, 2015

@ArnaudRinquin

This comment has been minimized.

Show comment
Hide comment
@ArnaudRinquin

ArnaudRinquin Apr 13, 2015

Contributor

Funny, mind blowing experience I had that made me realise how easy it is to fix this issue: I created a proof of concept project. npm install works perfectly. However, updating sub-modules versions in package.json only works for the my-module one... If you change the version, it will actually do an update. However, changes to my-other-module are just ignored. I spent hours trying to understand why.

This is why: https://www.npmjs.com/package/my-module actually exists.

So, because the request succeeds, the package is successfully updated, the way it's intended, regardless of what's returned.

I guess that the request should not be made if dependency is local, and just read the local package.json instead.

Contributor

ArnaudRinquin commented Apr 13, 2015

Funny, mind blowing experience I had that made me realise how easy it is to fix this issue: I created a proof of concept project. npm install works perfectly. However, updating sub-modules versions in package.json only works for the my-module one... If you change the version, it will actually do an update. However, changes to my-other-module are just ignored. I spent hours trying to understand why.

This is why: https://www.npmjs.com/package/my-module actually exists.

So, because the request succeeds, the package is successfully updated, the way it's intended, regardless of what's returned.

I guess that the request should not be made if dependency is local, and just read the local package.json instead.

@ArnaudRinquin

This comment has been minimized.

Show comment
Hide comment
@ArnaudRinquin

ArnaudRinquin Apr 13, 2015

Contributor

Actually, it's very easy to spot how npm outdated is broken on local modules:

$ npm outdated
Package    Current  Wanted  Latest  Location
my-module    2.0.0   2.1.0   1.0.0  my-module
  • Current version is the one from node_module/my-module/package.json
  • Wanted version is the one from path/to/local/my-module/package.json
  • Latest version is the one from https://www.npmjs.com/package/my-module, which doesn't make sense at all. It should be the same as wanted, right?

Also, because https://www.npmjs.com/package/my-other-module is a 404, it's completely ignored by the command.

Contributor

ArnaudRinquin commented Apr 13, 2015

Actually, it's very easy to spot how npm outdated is broken on local modules:

$ npm outdated
Package    Current  Wanted  Latest  Location
my-module    2.0.0   2.1.0   1.0.0  my-module
  • Current version is the one from node_module/my-module/package.json
  • Wanted version is the one from path/to/local/my-module/package.json
  • Latest version is the one from https://www.npmjs.com/package/my-module, which doesn't make sense at all. It should be the same as wanted, right?

Also, because https://www.npmjs.com/package/my-other-module is a 404, it's completely ignored by the command.

@ArnaudRinquin

This comment has been minimized.

Show comment
Hide comment
@ArnaudRinquin

ArnaudRinquin Apr 13, 2015

Contributor

From the tests, I understand that private packages (ie. local packages that are not published on the repository) are just expected to be ignored.

Why don't we just handle them?

This would solve the whole "Better local require() paths" problem so easily!

Contributor

ArnaudRinquin commented Apr 13, 2015

From the tests, I understand that private packages (ie. local packages that are not published on the repository) are just expected to be ignored.

Why don't we just handle them?

This would solve the whole "Better local require() paths" problem so easily!

@ArnaudRinquin

This comment has been minimized.

Show comment
Hide comment
@ArnaudRinquin

ArnaudRinquin Apr 13, 2015

Contributor

#7934 is ready and should make everybody happy.

Contributor

ArnaudRinquin commented Apr 13, 2015

#7934 is ready and should make everybody happy.

@williscool

This comment has been minimized.

Show comment
Hide comment
@williscool

williscool Apr 24, 2015

+1

Also here is what I don't understand.

Why when you use the file: directive in the package declaration

ala

    "ember-export-application-global": "^1.0.2",
    "spree-ember-core": "file:./submodules/spree-ember/packages/core/",
    "spree-ember-checkouts": "file:./submodules/spree-ember/packages/checkouts/",
    "spree-ember-storefront": "file:./submodules/spree-ember/packages/storefront/"

doesnt it just look for the packages at that directory?

Why does it require you to npm install and have npm install install them to the local node_modules dir and not track any updates?

I'm working on a box with several other people all working on the same project.

And while I'm the only one working on the modules I don't want to stomp on the global ones when im developing and breaking them and hold others up so npm link to the global directory isn't viable

+1

Also here is what I don't understand.

Why when you use the file: directive in the package declaration

ala

    "ember-export-application-global": "^1.0.2",
    "spree-ember-core": "file:./submodules/spree-ember/packages/core/",
    "spree-ember-checkouts": "file:./submodules/spree-ember/packages/checkouts/",
    "spree-ember-storefront": "file:./submodules/spree-ember/packages/storefront/"

doesnt it just look for the packages at that directory?

Why does it require you to npm install and have npm install install them to the local node_modules dir and not track any updates?

I'm working on a box with several other people all working on the same project.

And while I'm the only one working on the modules I don't want to stomp on the global ones when im developing and breaking them and hold others up so npm link to the global directory isn't viable

@othiym23

This comment has been minimized.

Show comment
Hide comment
@othiym23

othiym23 Apr 24, 2015

Contributor

Why when you use the file: directive in the package declaration ... doesn't it just look for the packages at that directory?

I'll give you a clue in the form of a question: what is 'it' in your question?

Node (and its module loader) doesn't know anything about npm or the contents of package.json except the contents of the main field. Conversely, npm isn't involved in the process of module loading at runtime. The Node module loader would have to change to support this feature, and while the module resolution and loading algorithm is looking less locked down now than it has in the past (thanks to io.js), it seems pretty unlikely that the Node core team is going to want to make Node more tightly coupled to npm rather than less.

As for why local dependencies aren't set up as symlinks in the first place, that's a question for #6251. Give linklocal a shot and see how it works for you. There hasn't been a lot of movement on that issue for a while, and the CLI's roadmap (in waffle form) is pretty full as-is.

Contributor

othiym23 commented Apr 24, 2015

Why when you use the file: directive in the package declaration ... doesn't it just look for the packages at that directory?

I'll give you a clue in the form of a question: what is 'it' in your question?

Node (and its module loader) doesn't know anything about npm or the contents of package.json except the contents of the main field. Conversely, npm isn't involved in the process of module loading at runtime. The Node module loader would have to change to support this feature, and while the module resolution and loading algorithm is looking less locked down now than it has in the past (thanks to io.js), it seems pretty unlikely that the Node core team is going to want to make Node more tightly coupled to npm rather than less.

As for why local dependencies aren't set up as symlinks in the first place, that's a question for #6251. Give linklocal a shot and see how it works for you. There hasn't been a lot of movement on that issue for a while, and the CLI's roadmap (in waffle form) is pretty full as-is.

@williscool

This comment has been minimized.

Show comment
Hide comment
@williscool

williscool Apr 24, 2015

@othiym23 Thanks that explanation makes a lot of since.

And yeah I just found linklocal it seems to suit my needs pretty well.

Just was really confusing at first because everything I had read up until your comment made it seem as if file: just made everything magically work.

@othiym23 Thanks that explanation makes a lot of since.

And yeah I just found linklocal it seems to suit my needs pretty well.

Just was really confusing at first because everything I had read up until your comment made it seem as if file: just made everything magically work.

rvagg added a commit to nodejs/node that referenced this issue May 4, 2015

2015-05-04 io.js v2.0.0 Release
PR-URL: #1532

Notable Changes:

* crypto: significantly reduced memory usage for TLS (Fedor Indutny & Сковорода
  Никита Андреевич) #1529
* net: socket.connect() now accepts a 'lookup' option for a custom DNS
  resolution mechanism, defaults to dns.lookup() (Evan Lucas) #1505
* npm: Upgrade npm to 2.9.0. See the v2.8.4 and v2.9.0 release notes for
  details. Notable items:
  - Add support for default author field to make npm init -y work without
    user-input (@othiym23) npm/npm/d8eee6cf9d
  - Include local modules in npm outdated and npm update (@ArnaudRinquin)
    npm/npm#7426
  - The prefix used before the version number on npm version is now configurable
    via tag-version-prefix (@kkragenbrink) npm/npm#8014
* os: os.tmpdir() is now cross-platform consistent and will no longer returns a
  path with a trailling slash on any platform (Christian Tellnes) #747
* process:
  - process.nextTick() performance has been improved by between 2-42% across the
    benchmark suite, notable because this is heavily used across core (Brian White) #1548
  - New process.geteuid(), process.seteuid(id), process.getegid() and
    process.setegid(id) methods allow you to get and set effective UID and GID
    of the process (Evan Lucas) #1536
* repl:
  - REPL history can be persisted across sessions if the NODE_REPL_HISTORY_FILE
    environment variable is set to a user accessible file,
    NODE_REPL_HISTORY_SIZE can set the maximum history size and defaults to 1000
    (Chris Dickinson) #1513
  - The REPL can be placed in to one of three modes using the NODE_REPL_MODE
    environment variable: sloppy, strict or magic (default); the new magic mode
    will automatically run "strict mode only" statements in strict mode
    (Chris Dickinson) #1513
* smalloc: the 'smalloc' module has been deprecated due to changes coming in V8
  4.4 that will render it unusable
* util: add Promise, Map and Set inspection support (Christopher Monsanto) #1471
* V8: upgrade to 4.2.77.18, see the ChangeLog for full details. Notable items:
  - Classes have moved out of staging; the class keyword is now usable in strict
    mode without flags
  - Object literal enhancements have moved out of staging; shorthand method and
    property syntax is now usable ({ method() { }, property })
  - Rest parameters (function(...args) {}) are implemented in staging behind the
    --harmony-rest-parameters flag
  - Computed property names ({['foo'+'bar']:'bam'}) are implemented in staging
    behind the --harmony-computed-property-names flag
  - Unicode escapes ('\u{xxxx}') are implemented in staging behind the
    --harmony_unicode flag and the --harmony_unicode_regexps flag for use in
    regular expressions
* Windows:
  - Random process termination on Windows fixed (Fedor Indutny) #1512 / #1563
  - The delay-load hook introduced to fix issues with process naming (iojs.exe /
    node.exe) has been made opt-out for native add-ons. Native add-ons should
    include 'win_delay_load_hook': 'false' in their binding.gyp to disable this
    feature if they experience problems . (Bert Belder) #1433
* Governance:
  - Rod Vagg (@rvagg) was added to the Technical Committee (TC)
  - Jeremiah Senkpiel (@Fishrock123) was added to the Technical Committee (TC)
@dazwin

This comment has been minimized.

Show comment
Hide comment
@dazwin

dazwin Aug 6, 2015

I came across this thread after having the same issue. My expectation is that a file: reference in the version field should work the same as any other type of dependency in the version field - i.e that an 'npm update' would pull the latest version within the given constraints (whatever packaging/unpacking is done internally). Just looking for consistency with http and git references.

I understand that npm link would give me a 'live' version of my dependent module, but this might not be what I want, and AFAICT is not supported by package.json.

dazwin commented Aug 6, 2015

I came across this thread after having the same issue. My expectation is that a file: reference in the version field should work the same as any other type of dependency in the version field - i.e that an 'npm update' would pull the latest version within the given constraints (whatever packaging/unpacking is done internally). Just looking for consistency with http and git references.

I understand that npm link would give me a 'live' version of my dependent module, but this might not be what I want, and AFAICT is not supported by package.json.

@akiva

This comment has been minimized.

Show comment
Hide comment
@akiva

akiva Sep 8, 2015

+1

I, too, would love to see a working solution native to npm here. I commented on my use cases and rationale for my hopes as comments here:
https://gist.github.com/branneman/8048520#gistcomment-1567153 and https://gist.github.com/branneman/8048520#gistcomment-1567162

akiva commented Sep 8, 2015

+1

I, too, would love to see a working solution native to npm here. I commented on my use cases and rationale for my hopes as comments here:
https://gist.github.com/branneman/8048520#gistcomment-1567153 and https://gist.github.com/branneman/8048520#gistcomment-1567162

@kossnocorp

This comment has been minimized.

Show comment
Hide comment
@kossnocorp

kossnocorp Oct 4, 2015

Contributor

There another flaw in the first approach: if you want to have your package's root directory as a link target then you will end-up in infinite loop of postinstall runs.

Solution is to remove the symlink in preinstall:

"scripts": {
  "preinstall": "rm -f $(pwd)/node_modules/app",
  "postinstall": "mkdir -p $(pwd)/node_modules && ln -s $(pwd) $(pwd)/node_modules/app"
}

* it's not adapted for Windows
** mkdir -p $(pwd)/node_modules is needed because package may have 0 dependencies.

Contributor

kossnocorp commented Oct 4, 2015

There another flaw in the first approach: if you want to have your package's root directory as a link target then you will end-up in infinite loop of postinstall runs.

Solution is to remove the symlink in preinstall:

"scripts": {
  "preinstall": "rm -f $(pwd)/node_modules/app",
  "postinstall": "mkdir -p $(pwd)/node_modules && ln -s $(pwd) $(pwd)/node_modules/app"
}

* it's not adapted for Windows
** mkdir -p $(pwd)/node_modules is needed because package may have 0 dependencies.

@wavded

This comment has been minimized.

Show comment
Hide comment
@wavded

wavded Oct 15, 2015

This is likely a separate issue so I can make a case if deemed so, but @kossnocorp's solution (which is very similar to what I use) fails on npm@3 when trying to shrinkwrap (doesn't on npm@2) because the symlink is an "extraneous dependency". Not sure how to work around that yet or if that is a bug.

wavded commented Oct 15, 2015

This is likely a separate issue so I can make a case if deemed so, but @kossnocorp's solution (which is very similar to what I use) fails on npm@3 when trying to shrinkwrap (doesn't on npm@2) because the symlink is an "extraneous dependency". Not sure how to work around that yet or if that is a bug.

@kossnocorp

This comment has been minimized.

Show comment
Hide comment
@kossnocorp

kossnocorp Oct 15, 2015

Contributor

@wavded didn't tried, but I'm pretty sure that you can do such trick:

"preshrinkwrap": "npm run prebuild",
"postshrinkwrap": "npm run postbuild",
Contributor

kossnocorp commented Oct 15, 2015

@wavded didn't tried, but I'm pretty sure that you can do such trick:

"preshrinkwrap": "npm run prebuild",
"postshrinkwrap": "npm run postbuild",
@wavded

This comment has been minimized.

Show comment
Hide comment
@wavded

wavded Oct 30, 2015

@kossnocorp seems like there isn't lifecycle hooks for shrinkwrap, at least i haven't had any luck so far

wavded commented Oct 30, 2015

@kossnocorp seems like there isn't lifecycle hooks for shrinkwrap, at least i haven't had any luck so far

@jordwalke

This comment has been minimized.

Show comment
Hide comment
@jordwalke

jordwalke Nov 25, 2015

I'm surprised that this does not work, and I'm really missing this feature. Here is the use case:

  • I have one private code repo that contains many smaller npm projects.
  • There is a root project that depends on all the other ones.
  • I want the instructions for installing dependencies to be cd rootDir && npm install.
  • I want the instructions for updating after rebasing the project to be cd rootDir && npm update.

It should really be that easy. Here are some things I tried and why they fail:

  1. I tried local dependencies with file:../blah. But it copies the entire dependency into node_modules and so won't be kept up to date when making local changes. It also doesn't update with npm update from the root package if one of the dependencies now needs a new downloaded dependency.
  2. Symlinks are linked correctly, and code changes in dependencies are thankfully always represented - but they are not updated when running npm update either.
  3. If you create a separate networked repo for each dependency, and point to a particular git hash for each one, npm update works but is overly aggressive in fetching updates. (And sometimes you can't just go creating a new private hosted git repo).

I feel like npm is sooo close to being the killer local development manager, but misses one tiny, but critical feature for each of (git urls, sym links, and local dependency paths).

I'm surprised that this does not work, and I'm really missing this feature. Here is the use case:

  • I have one private code repo that contains many smaller npm projects.
  • There is a root project that depends on all the other ones.
  • I want the instructions for installing dependencies to be cd rootDir && npm install.
  • I want the instructions for updating after rebasing the project to be cd rootDir && npm update.

It should really be that easy. Here are some things I tried and why they fail:

  1. I tried local dependencies with file:../blah. But it copies the entire dependency into node_modules and so won't be kept up to date when making local changes. It also doesn't update with npm update from the root package if one of the dependencies now needs a new downloaded dependency.
  2. Symlinks are linked correctly, and code changes in dependencies are thankfully always represented - but they are not updated when running npm update either.
  3. If you create a separate networked repo for each dependency, and point to a particular git hash for each one, npm update works but is overly aggressive in fetching updates. (And sometimes you can't just go creating a new private hosted git repo).

I feel like npm is sooo close to being the killer local development manager, but misses one tiny, but critical feature for each of (git urls, sym links, and local dependency paths).

@kentingxu

This comment has been minimized.

Show comment
Hide comment
@kentingxu

kentingxu Jan 10, 2016

I have the same problem when using local dependencies. npm update doesn't update my local dependencies even I increase the version number in my module's package.json file.
After spent a night to try different things, finally I figure out if I remove "private": true from my module's package.json file, npm update CAN do the update. I don't understand why npm update ignore private modules, or it is just a bug?

I'm using
npm 3.3.12
node v5.1.1

I have the same problem when using local dependencies. npm update doesn't update my local dependencies even I increase the version number in my module's package.json file.
After spent a night to try different things, finally I figure out if I remove "private": true from my module's package.json file, npm update CAN do the update. I don't understand why npm update ignore private modules, or it is just a bug?

I'm using
npm 3.3.12
node v5.1.1

@myme

This comment has been minimized.

Show comment
Hide comment
@myme

myme Jan 12, 2016

I'm seing the same thing as @kentingxu. It's very convenient to list modules as local fs dependencies of a parent project and use npm install and npm update to manage them. It makes sense for us to set the private flag though, as they are only relevant to our company and shouldn't be published to any npm registry.

❯ npm --version
3.3.12
❯ node --version
v5.2.0

myme commented Jan 12, 2016

I'm seing the same thing as @kentingxu. It's very convenient to list modules as local fs dependencies of a parent project and use npm install and npm update to manage them. It makes sense for us to set the private flag though, as they are only relevant to our company and shouldn't be published to any npm registry.

❯ npm --version
3.3.12
❯ node --version
v5.2.0
@hamxiaoz

This comment has been minimized.

Show comment
Hide comment
@hamxiaoz

hamxiaoz Mar 3, 2016

I have the same problem, my workaround for now is to just delete the package folder in node_modules then do npm install again. Silly but works...

hamxiaoz commented Mar 3, 2016

I have the same problem, my workaround for now is to just delete the package folder in node_modules then do npm install again. Silly but works...

@gartenfeld

This comment has been minimized.

Show comment
Hide comment
@gartenfeld

gartenfeld Apr 18, 2016

This is still a bug in 3.8.2.
Neither npm install nor npm install --force nor npm update fetches changes from a repo-based module source. The only way to get the changes is to remove the module directory first.

This is still a bug in 3.8.2.
Neither npm install nor npm install --force nor npm update fetches changes from a repo-based module source. The only way to get the changes is to remove the module directory first.

@chyzwar

This comment has been minimized.

Show comment
Hide comment
@chyzwar

chyzwar Apr 19, 2016

I have the same issue. It is a bug. Local packages should be treated the same as any other.

chyzwar commented Apr 19, 2016

I have the same issue. It is a bug. Local packages should be treated the same as any other.

@mbardelmeijer

This comment has been minimized.

Show comment
Hide comment
@mbardelmeijer

mbardelmeijer Apr 30, 2016

Same issue here. npm install should work regardless if it's a local package or not. Only solution now is to remove the node_modules directory :(

Same issue here. npm install should work regardless if it's a local package or not. Only solution now is to remove the node_modules directory :(

@UziTech

This comment has been minimized.

Show comment
Hide comment
@UziTech

UziTech Jun 22, 2016

My solution is to namespace my modules so I only have to remove the namespaced modules instead of the whole node_modules folder

  "scripts": {
    "preinstall": "rimraf node_modules/namespace-*"
  }

UziTech commented Jun 22, 2016

My solution is to namespace my modules so I only have to remove the namespaced modules instead of the whole node_modules folder

  "scripts": {
    "preinstall": "rimraf node_modules/namespace-*"
  }
@rickyo-CI

This comment has been minimized.

Show comment
Hide comment
@rickyo-CI

rickyo-CI Mar 23, 2017

Is this still an issue? I'm having the sam problem with local packages. removing the node_modules directory as a workaround gets annoying before installing. Are there any other external npm packages that can do this?

Is this still an issue? I'm having the sam problem with local packages. removing the node_modules directory as a workaround gets annoying before installing. Are there any other external npm packages that can do this?

@josh08h

This comment has been minimized.

Show comment
Hide comment
@josh08h

josh08h Apr 22, 2017

I still have the issue that running npm update in a package that has local package dependencies, that have moved to a new version, does nothing. The rm -rf node_modules can be tedious. Is the only solution to symlink?

josh08h commented Apr 22, 2017

I still have the issue that running npm update in a package that has local package dependencies, that have moved to a new version, does nothing. The rm -rf node_modules can be tedious. Is the only solution to symlink?

@xkr47

This comment has been minimized.

Show comment
Hide comment

xkr47 commented Apr 24, 2017

See #10379 and #16013.

@cdhowie

This comment has been minimized.

Show comment
Hide comment
@cdhowie

cdhowie May 11, 2017

Using npm upgrade to update local (file:foo) dependencies does seem to work in npm 3.10.10, at least for me (after bumping the version number in foo/package.json).

cdhowie commented May 11, 2017

Using npm upgrade to update local (file:foo) dependencies does seem to work in npm 3.10.10, at least for me (after bumping the version number in foo/package.json).

@xkr47

This comment has been minimized.

Show comment
Hide comment
@xkr47

xkr47 May 11, 2017

Is the behaviour for e.g. github repos the same?

  • install does nothing
  • upgrade does nothing if version doesn't change
  • upgrade updates if version changed

xkr47 commented May 11, 2017

Is the behaviour for e.g. github repos the same?

  • install does nothing
  • upgrade does nothing if version doesn't change
  • upgrade updates if version changed
@cdhowie

This comment has been minimized.

Show comment
Hide comment
@cdhowie

cdhowie May 15, 2017

@xkr47 I did some cursory tests and it appears that upgrade always reinstalls modules from github regardless if the version number changed.

cdhowie commented May 15, 2017

@xkr47 I did some cursory tests and it appears that upgrade always reinstalls modules from github regardless if the version number changed.

@npm-robot

This comment has been minimized.

Show comment
Hide comment
@npm-robot

npm-robot Jun 19, 2017

We're closing this issue as it has gone thirty days without activity. In our experience if an issue has gone thirty days without any activity then it's unlikely to be addressed. In the case of bug reports, often the underlying issue will be addressed but finding related issues is quite difficult and often incomplete.

If this was a bug report and it is still relevant then we encourage you to open it again as a new issue. If this was a feature request then you should feel free to open it again, or even better open a PR.

For more information about our new issue aging policies and why we've instituted them please see our blog post.

We're closing this issue as it has gone thirty days without activity. In our experience if an issue has gone thirty days without any activity then it's unlikely to be addressed. In the case of bug reports, often the underlying issue will be addressed but finding related issues is quite difficult and often incomplete.

If this was a bug report and it is still relevant then we encourage you to open it again as a new issue. If this was a feature request then you should feel free to open it again, or even better open a PR.

For more information about our new issue aging policies and why we've instituted them please see our blog post.

@vincelord

This comment has been minimized.

Show comment
Hide comment
@vincelord

vincelord Aug 23, 2017

... aaaaaand reopened

... aaaaaand reopened

@kimek

This comment has been minimized.

Show comment
Hide comment
@kimek

kimek Oct 19, 2017

npm cache clean ( Optional step)
rm -r /node_module/{{your-package}}
npm up {{your-package}}
Works for getting last git repos without tags for us

kimek commented Oct 19, 2017

npm cache clean ( Optional step)
rm -r /node_module/{{your-package}}
npm up {{your-package}}
Works for getting last git repos without tags for us

@akiva

This comment has been minimized.

Show comment
Hide comment
@akiva

akiva Oct 19, 2017

@kimek While there are many workarounds, I believe the larger issue here is the API that is baked into npm isn't working as expected for this use case.

akiva commented Oct 19, 2017

@kimek While there are many workarounds, I believe the larger issue here is the API that is baked into npm isn't working as expected for this use case.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.