Update of local dependencies does not work

[akloeber]

I have some local dependencies on sub-modules in the same git repo like this:

"dependencies": {
  "sub-mod-0": "./sub-mod-0",
  "sub-mod-1": "./sub-mod-1"
}

With this I can do require('sub-mod-0') without having to use something like require('../../sub-mod-0')

Installation via npm install works fine and populates node_modules/sub-mod-0 and node_modules/sub-mod-1 but updating them is not possible without manually deleting node_modules/sub-mod-* beforehand. Neither npm install nor npm update nor npm update sub-module-* have an effect even if the version of the sub-module is incremented.

Environment:
npm v2.6.0
node v0.10.25
Ubuntu 14.04.1 LTS

[smikes]

I think you may want to use npm link, if what you're hoping to see is live updates of your dependencies in your parent module.

When you depend on a directory via npm install ../whatever, npm actually builds a tarball based on the state at the time of npm install, stores that tarball in the cache, and then installs that tarball from cache, so (as you note) you are not seeing the live dependencies.

With npm link, npm creates a link from its global install location (see prefix) that points to the dependency; then when you link the dependency into your package, npm makes a link from node_modules that points at the target.

In general this is recommended for development, not production though. Can you tell me more about your situation and what you're trying to accomplish?

See https://docs.npmjs.com/cli/link
and https://docs.npmjs.com/files/folders

[akloeber]

Well, let me clarify our situation:
We have a larger system that consists of several "components" that use each other. First we had linked them together directly through file system traversal (i.e. require('../../some-module')) which was not very stable across refactorings because when moving things around the paths had to be adjusted.

So we tried to store each sub-module in its own repository and require() them in the parent module as normal. But it appeared that there were often changes/features across sub module (i.e. component) boundaries and having them in different repos was impractible as these had to be split over several commits in different repositories.

So we aggregated them into a single repository again and used local dependencies once they were introduced. That resolved our problems except the issue during deployment described above (currently we work around that by deleting node_modules followed by npm install which is very time-consuming).

We already know about npm link and use it during development. We do not want to use in production as it is not recommended and we want to avoid the extra scripting for the npm link steps.

What would be the right approach in this case?

[smikes]

What about setting up your NODE_PATH in production so that it searches all of your submodule roots?

http://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

Another option would be to write a wrapper around require but I think you can get what you want just by using NODE_PATH...

[akloeber]

Using NODE_PATH would work but then we still have to do the scripting for executing npm install in every sub-module directory and the content of NODE_PATH has to be managed to keep it in sync with the sub-modules. With the current approach a single execution of npm install populates everything.

We already discussed writing a wrapper/replacement for require but that seems to be a hack and we are still looking for a clean solution.

[smikes]

This is getting well beyond my expertise unfortunately -- it is essentially a devops question now. One thing that might work for deployment would be to build the entire package tree once, offline, and pack it up with "npm pack" (possibly using bundledDependencies)

Then you would have a single tgz representing your new version and could deploy it across however many instances you are running.

If you have to support upgrading in-place, then "rm -rf node_modules; npm install" is the best way to be sure... How long is "very time-consuming"?

And of course npm@3 should help with this, because its approach is to build the complete desired dependency graph, calculate the difference between desired and current, and then apply the changes needed to make the transition. Should be cheaper than what you see now.

[akloeber]

Yes, I suppose uploading a tgz and extracting it might save some time in comparison to rm -rf node_modules; npm install but therefore adds a new build step. When I say "very time-consuming" I mean several minutes which is still OK for production deployment but suboptimal for our CI environment.

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I was hoping for something like npm update --force. Another approach might be optional dependency tracking options regarded by npm update to support fine-tuning for these use cases like:

"dependencies": {
  "lodash": "^2.4.1",
  "sub-mod-0": {
    "path": "./sub-mod-0",
    "tracking": "ALWAYS_SYNC"
  },
  "sub-mod-1": "./sub-mod-1"
}

Nevertheless: Thank you very much for your support and your detailed explanations!

[smikes]

Here's the blog post that lays out the roadmap for npm@3: http://blog.npmjs.org/post/98233700815/multi-stage-installs-and-a-better-npm

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I think that's not quite right; it's even worse than that. If you are directly installing from git submodules then npm treats them as pretty opaque and does not look into them to do version management. So npm update won't update a git dependency, and you'll have to rely on npm install. It doesn't matter if the commit-ish part of the dependency changes, or if the version in package.json changes.

You may be able to mitigate this a bit by running a private npm registry where you publish your own scoped and versioned modules. Then you can use npm install of prepackaged tarballs in your CI, which should be somewhat faster.

Also you might want to check out npm's paid product, npm Enterprise – I'm not an employee of npm, inc., and I don't actually know precisely what it does. But it may help addressing some of these issues.

[akloeber]

As far as I understand, at least npm install "updates" git dependencies to the current head. Is that right? (see #1727 and #4104)

I looked at the feature list of npm Enterprise but I could not find anything that might help. A private registry with scoped modules would be similar to our first approach when we used different repos for each submodule. If my assumption on the top is correct we can already achieve by using git URLs to dedicated repos that each contain a submodule. But as I described our modules are not that independent.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install). If it is another invocation of npm install then it should update packages from the registry based on their version, from git based on the git meta data and sync local resources as there is not meta data.

[akloeber]

I found a longer discussion on #1558 with similar use cases and the local paths feature seems to be the "official" fix for those scenarios. If npm install does not update dependencies with local paths then some of them would not be fixed neither.

There is also the suggestion of using preinstall: "npm install sub-module-*" or preinstall: "npm link sub-module-* --prefix=node_modules" in package.json but they are still a kind of workaround.

[smikes]

wrt npm-enterprise, as I said, I don't know much about the product.

I was thinking it may be faster/ more efficient if your sub-dependencies were already packaged as modules. An npm install from a git repository consists of two clones, a pack, and an install, while npm install from an npm repository is just a download + install, so installing from a prebuilt tarball should be faster.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install).

I don't believe there is a preferred way in npm@2; or rather, "rm -rf node_modules && npm install" is the most succinct way to ensure that this happens

The multi-stage-install core feature of npm@3 has been in development for several months, in order to address the problem of taking the difference between an installation and a specification, and implementing the steps to correct the difference.

[smikes]

You may want to look at #6251 (unfulfilled feature request)

On Wed, Feb 25, 2015 at 10:04 AM, Andreas Klöber notifications@github.com
wrote:

I found a longer discussion on #1558
#1558 with similar use cases and the
local paths feature seems to be the "official" fix for those scenarios. If npm
install does not update dependencies with local paths then some of them
would not be fixed neither.

There is also the suggestion of using preinstall: "npm install
sub-module-" or preinstall: "npm link sub-module- --prefix=node_modules"
in package.json but they are still a kind of workaround.

—
Reply to this email directly or view it on GitHub
#7426 (comment).

[othiym23]

I believe this scenario is one of many that will be handled much better by npm@3, which is much more aggressive about making sure that dependencies are brought up to date each time npm install is run, but I don't know how much attention @iarna has given to local dependencies on her work thus far. If she has time, she might drop a note here saying where that's at, or I might take a look at it myself, as the person most responsible for local dependencies in their current state.

[iarna]

So with npm@2 I see it updating the local modules. With npm@3 it does not, as the spec from the dependencies in the package.json is met by the version already installed.

However, this does seem like a legitimate bug in npm outdated and thus npm update.

[ArnaudRinquin]

Funny, mind blowing experience I had that made me realise how easy it is to fix this issue: I created a proof of concept project. npm install works perfectly. However, updating sub-modules versions in package.json only works for the my-module one... If you change the version, it will actually do an update. However, changes to my-other-module are just ignored. I spent hours trying to understand why.

This is why: https://www.npmjs.com/package/my-module actually exists.

So, because the request succeeds, the package is successfully updated, the way it's intended, regardless of what's returned.

I guess that the request should not be made if dependency is local, and just read the local package.json instead.

[ArnaudRinquin]

Actually, it's very easy to spot how npm outdated is broken on local modules:

$ npm outdated
Package    Current  Wanted  Latest  Location
my-module    2.0.0   2.1.0   1.0.0  my-module

• Current version is the one from node_module/my-module/package.json
• Wanted version is the one from path/to/local/my-module/package.json
• Latest version is the one from https://www.npmjs.com/package/my-module, which doesn't make sense at all. It should be the same as wanted, right?

Also, because https://www.npmjs.com/package/my-other-module is a 404, it's completely ignored by the command.

[ArnaudRinquin]

From the tests, I understand that private packages (ie. local packages that are not published on the repository) are just expected to be ignored.

Why don't we just handle them?

This would solve the whole "Better local require() paths" problem so easily!

[ArnaudRinquin]

#7934 is ready and should make everybody happy.

[williscool]

+1

Also here is what I don't understand.

Why when you use the file: directive in the package declaration

ala

    "ember-export-application-global": "^1.0.2",
    "spree-ember-core": "file:./submodules/spree-ember/packages/core/",
    "spree-ember-checkouts": "file:./submodules/spree-ember/packages/checkouts/",
    "spree-ember-storefront": "file:./submodules/spree-ember/packages/storefront/"

doesnt it just look for the packages at that directory?

Why does it require you to npm install and have npm install install them to the local node_modules dir and not track any updates?

I'm working on a box with several other people all working on the same project.

And while I'm the only one working on the modules I don't want to stomp on the global ones when im developing and breaking them and hold others up so npm link to the global directory isn't viable

[othiym23]

Why when you use the file: directive in the package declaration ... doesn't it just look for the packages at that directory?

I'll give you a clue in the form of a question: what is 'it' in your question?

Node (and its module loader) doesn't know anything about npm or the contents of package.json except the contents of the main field. Conversely, npm isn't involved in the process of module loading at runtime. The Node module loader would have to change to support this feature, and while the module resolution and loading algorithm is looking less locked down now than it has in the past (thanks to io.js), it seems pretty unlikely that the Node core team is going to want to make Node more tightly coupled to npm rather than less.

As for why local dependencies aren't set up as symlinks in the first place, that's a question for #6251. Give linklocal a shot and see how it works for you. There hasn't been a lot of movement on that issue for a while, and the CLI's roadmap (in waffle form) is pretty full as-is.

[williscool]

@othiym23 Thanks that explanation makes a lot of since.

And yeah I just found linklocal it seems to suit my needs pretty well.

Just was really confusing at first because everything I had read up until your comment made it seem as if file: just made everything magically work.

[dazwin]

I came across this thread after having the same issue. My expectation is that a file: reference in the version field should work the same as any other type of dependency in the version field - i.e that an 'npm update' would pull the latest version within the given constraints (whatever packaging/unpacking is done internally). Just looking for consistency with http and git references.

I understand that npm link would give me a 'live' version of my dependent module, but this might not be what I want, and AFAICT is not supported by package.json.

[akiva]

+1

I, too, would love to see a working solution native to npm here. I commented on my use cases and rationale for my hopes as comments here:
https://gist.github.com/branneman/8048520#gistcomment-1567153 and https://gist.github.com/branneman/8048520#gistcomment-1567162

[kossnocorp]

There another flaw in the first approach: if you want to have your package's root directory as a link target then you will end-up in infinite loop of postinstall runs.

Solution is to remove the symlink in preinstall:

"scripts": {
  "preinstall": "rm -f $(pwd)/node_modules/app",
  "postinstall": "mkdir -p $(pwd)/node_modules && ln -s $(pwd) $(pwd)/node_modules/app"
}

* it's not adapted for Windows
** mkdir -p $(pwd)/node_modules is needed because package may have 0 dependencies.

[wavded]

This is likely a separate issue so I can make a case if deemed so, but @kossnocorp's solution (which is very similar to what I use) fails on npm@3 when trying to shrinkwrap (doesn't on npm@2) because the symlink is an "extraneous dependency". Not sure how to work around that yet or if that is a bug.

[kossnocorp]

@wavded didn't tried, but I'm pretty sure that you can do such trick:

"preshrinkwrap": "npm run prebuild",
"postshrinkwrap": "npm run postbuild",

[wavded]

@kossnocorp seems like there isn't lifecycle hooks for shrinkwrap, at least i haven't had any luck so far

[jordwalke]

I'm surprised that this does not work, and I'm really missing this feature. Here is the use case:

• I have one private code repo that contains many smaller npm projects.
• There is a root project that depends on all the other ones.
• I want the instructions for installing dependencies to be cd rootDir && npm install.
• I want the instructions for updating after rebasing the project to be cd rootDir && npm update.

It should really be that easy. Here are some things I tried and why they fail:

1. I tried local dependencies with file:../blah. But it copies the entire dependency into node_modules and so won't be kept up to date when making local changes. It also doesn't update with npm update from the root package if one of the dependencies now needs a new downloaded dependency.
2. Symlinks are linked correctly, and code changes in dependencies are thankfully always represented - but they are not updated when running npm update either.
3. If you create a separate networked repo for each dependency, and point to a particular git hash for each one, npm update works but is overly aggressive in fetching updates. (And sometimes you can't just go creating a new private hosted git repo).

I feel like npm is sooo close to being the killer local development manager, but misses one tiny, but critical feature for each of (git urls, sym links, and local dependency paths).

[kentingxu]

I have the same problem when using local dependencies. npm update doesn't update my local dependencies even I increase the version number in my module's package.json file.
After spent a night to try different things, finally I figure out if I remove "private": true from my module's package.json file, npm update CAN do the update. I don't understand why npm update ignore private modules, or it is just a bug?

I'm using
npm 3.3.12
node v5.1.1

[myme]

I'm seing the same thing as @kentingxu. It's very convenient to list modules as local fs dependencies of a parent project and use npm install and npm update to manage them. It makes sense for us to set the private flag though, as they are only relevant to our company and shouldn't be published to any npm registry.

❯ npm --version
3.3.12
❯ node --version
v5.2.0

[hamxiaoz]

I have the same problem, my workaround for now is to just delete the package folder in node_modules then do npm install again. Silly but works...

[gartenfeld]

This is still a bug in 3.8.2.
Neither npm install nor npm install --force nor npm update fetches changes from a repo-based module source. The only way to get the changes is to remove the module directory first.

[chyzwar]

I have the same issue. It is a bug. Local packages should be treated the same as any other.

[mbardelmeijer]

Same issue here. npm install should work regardless if it's a local package or not. Only solution now is to remove the node_modules directory :(

[UziTech]

My solution is to namespace my modules so I only have to remove the namespaced modules instead of the whole node_modules folder

  "scripts": {
    "preinstall": "rimraf node_modules/namespace-*"
  }

[rickyo-CI]

Is this still an issue? I'm having the sam problem with local packages. removing the node_modules directory as a workaround gets annoying before installing. Are there any other external npm packages that can do this?

[josh08h]

I still have the issue that running npm update in a package that has local package dependencies, that have moved to a new version, does nothing. The rm -rf node_modules can be tedious. Is the only solution to symlink?

[xkr47]

See #10379 and #16013.

[cdhowie]

Using npm upgrade to update local (file:foo) dependencies does seem to work in npm 3.10.10, at least for me (after bumping the version number in foo/package.json).

[xkr47]

Is the behaviour for e.g. github repos the same?

• install does nothing
• upgrade does nothing if version doesn't change
• upgrade updates if version changed

[cdhowie]

@xkr47 I did some cursory tests and it appears that upgrade always reinstalls modules from github regardless if the version number changed.

[npm-robot]

We're closing this issue as it has gone thirty days without activity. In our experience if an issue has gone thirty days without any activity then it's unlikely to be addressed. In the case of bug reports, often the underlying issue will be addressed but finding related issues is quite difficult and often incomplete.

If this was a bug report and it is still relevant then we encourage you to open it again as a new issue. If this was a feature request then you should feel free to open it again, or even better open a PR.

For more information about our new issue aging policies and why we've instituted them please see our blog post.

[vincolus]

... aaaaaand reopened

[kimek]

npm cache clean ( Optional step)
rm -r /node_module/{{your-package}}
npm up {{your-package}}
Works for getting last git repos without tags for us

[akiva]

@kimek While there are many workarounds, I believe the larger issue here is the API that is baked into npm isn't working as expected for this use case.