Update of local dependencies does not work

[akloeber]

I have some local dependencies on sub-modules in the same git repo like this:

"dependencies": {
  "sub-mod-0": "./sub-mod-0",
  "sub-mod-1": "./sub-mod-1"
}

With this I can do require('sub-mod-0') without having to use something like require('../../sub-mod-0')

Installation via npm install works fine and populates node_modules/sub-mod-0 and node_modules/sub-mod-1 but updating them is not possible without manually deleting node_modules/sub-mod-* beforehand. Neither npm install nor npm update nor npm update sub-module-* have an effect even if the version of the sub-module is incremented.

Environment:
npm v2.6.0
node v0.10.25
Ubuntu 14.04.1 LTS

[smikes]

I think you may want to use npm link, if what you're hoping to see is live updates of your dependencies in your parent module.

When you depend on a directory via npm install ../whatever, npm actually builds a tarball based on the state at the time of npm install, stores that tarball in the cache, and then installs that tarball from cache, so (as you note) you are not seeing the live dependencies.

With npm link, npm creates a link from its global install location (see prefix) that points to the dependency; then when you link the dependency into your package, npm makes a link from node_modules that points at the target.

In general this is recommended for development, not production though. Can you tell me more about your situation and what you're trying to accomplish?

See https://docs.npmjs.com/cli/link
and https://docs.npmjs.com/files/folders

[akloeber]

Well, let me clarify our situation:
We have a larger system that consists of several "components" that use each other. First we had linked them together directly through file system traversal (i.e. require('../../some-module')) which was not very stable across refactorings because when moving things around the paths had to be adjusted.

So we tried to store each sub-module in its own repository and require() them in the parent module as normal. But it appeared that there were often changes/features across sub module (i.e. component) boundaries and having them in different repos was impractible as these had to be split over several commits in different repositories.

So we aggregated them into a single repository again and used local dependencies once they were introduced. That resolved our problems except the issue during deployment described above (currently we work around that by deleting node_modules followed by npm install which is very time-consuming).

We already know about npm link and use it during development. We do not want to use in production as it is not recommended and we want to avoid the extra scripting for the npm link steps.

What would be the right approach in this case?

[smikes]

What about setting up your NODE_PATH in production so that it searches all of your submodule roots?

http://nodejs.org/api/modules.html#modules_loading_from_the_global_folders

Another option would be to write a wrapper around require but I think you can get what you want just by using NODE_PATH...

[akloeber]

Using NODE_PATH would work but then we still have to do the scripting for executing npm install in every sub-module directory and the content of NODE_PATH has to be managed to keep it in sync with the sub-modules. With the current approach a single execution of npm install populates everything.

We already discussed writing a wrapper/replacement for require but that seems to be a hack and we are still looking for a clean solution.

[smikes]

This is getting well beyond my expertise unfortunately -- it is essentially a devops question now. One thing that might work for deployment would be to build the entire package tree once, offline, and pack it up with "npm pack" (possibly using bundledDependencies)

Then you would have a single tgz representing your new version and could deploy it across however many instances you are running.

If you have to support upgrading in-place, then "rm -rf node_modules; npm install" is the best way to be sure... How long is "very time-consuming"?

And of course npm@3 should help with this, because its approach is to build the complete desired dependency graph, calculate the difference between desired and current, and then apply the changes needed to make the transition. Should be cheaper than what you see now.

[akloeber]

Yes, I suppose uploading a tgz and extracting it might save some time in comparison to rm -rf node_modules; npm install but therefore adds a new build step. When I say "very time-consuming" I mean several minutes which is still OK for production deployment but suboptimal for our CI environment.

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I was hoping for something like npm update --force. Another approach might be optional dependency tracking options regarded by npm update to support fine-tuning for these use cases like:

"dependencies": {
  "lodash": "^2.4.1",
  "sub-mod-0": {
    "path": "./sub-mod-0",
    "tracking": "ALWAYS_SYNC"
  },
  "sub-mod-1": "./sub-mod-1"
}

Nevertheless: Thank you very much for your support and your detailed explanations!

[smikes]

Here's the blog post that lays out the roadmap for npm@3: http://blog.npmjs.org/post/98233700815/multi-stage-installs-and-a-better-npm

Is there a link where the new dependency management of npm@3 is described? It sounds interesting but I reckon that it is based on the module versions and not on the content. So if the version sub-module version is not incremented the content is not synchronized. Is that right?

I think that's not quite right; it's even worse than that. If you are directly installing from git submodules then npm treats them as pretty opaque and does not look into them to do version management. So npm update won't update a git dependency, and you'll have to rely on npm install. It doesn't matter if the commit-ish part of the dependency changes, or if the version in package.json changes.

You may be able to mitigate this a bit by running a private npm registry where you publish your own scoped and versioned modules. Then you can use npm install of prepackaged tarballs in your CI, which should be somewhat faster.

Also you might want to check out npm's paid product, npm Enterprise – I'm not an employee of npm, inc., and I don't actually know precisely what it does. But it may help addressing some of these issues.

[akloeber]

As far as I understand, at least npm install "updates" git dependencies to the current head. Is that right? (see #1727 and #4104)

I looked at the feature list of npm Enterprise but I could not find anything that might help. A private registry with scoped modules would be similar to our first approach when we used different repos for each submodule. If my assumption on the top is correct we can already achieve by using git URLs to dedicated repos that each contain a submodule. But as I described our modules are not that independent.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install). If it is another invocation of npm install then it should update packages from the registry based on their version, from git based on the git meta data and sync local resources as there is not meta data.

[akloeber]

I found a longer discussion on #1558 with similar use cases and the local paths feature seems to be the "official" fix for those scenarios. If npm install does not update dependencies with local paths then some of them would not be fixed neither.

There is also the suggestion of using preinstall: "npm install sub-module-*" or preinstall: "npm link sub-module-* --prefix=node_modules" in package.json but they are still a kind of workaround.

[smikes]

wrt npm-enterprise, as I said, I don't know much about the product.

I was thinking it may be faster/ more efficient if your sub-dependencies were already packaged as modules. An npm install from a git repository consists of two clones, a pack, and an install, while npm install from an npm repository is just a download + install, so installing from a prebuilt tarball should be faster.

I'm still wondering what the preferred npm-way for updating a previously populated package structure is (instead of rm -rf node_modules && npm install).

I don't believe there is a preferred way in npm@2; or rather, "rm -rf node_modules && npm install" is the most succinct way to ensure that this happens

The multi-stage-install core feature of npm@3 has been in development for several months, in order to address the problem of taking the difference between an installation and a specification, and implementing the steps to correct the difference.

[smikes]

You may want to look at #6251 (unfulfilled feature request)

On Wed, Feb 25, 2015 at 10:04 AM, Andreas Klöber notifications@github.com
wrote:

I found a longer discussion on #1558
#1558 with similar use cases and the
local paths feature seems to be the "official" fix for those scenarios. If npm
install does not update dependencies with local paths then some of them
would not be fixed neither.

There is also the suggestion of using preinstall: "npm install
sub-module-" or preinstall: "npm link sub-module- --prefix=node_modules"
in package.json but they are still a kind of workaround.

—
Reply to this email directly or view it on GitHub
#7426 (comment).