local private module dependencies

[tj]

hey hey, looking for your thoughts or suggestions on this issue. What I wanted ideally is to modularize our app(s) using vanilla node_modules, however they are not in any registry so they're checked into the repo. $ npm install see's that they're already installed and ignores them, even if their own dependencies might not be installed. I have bigger plans for this so ideally avoiding a shell script to iterate and $ cd node_modules/pkg && npm install if possible.

Any suggestions for this use-case? maybe if "private": true is specified (or similar) npm could check the deps.

[mmalecki]

I think npm should just iterate over bundledDependencies and run npm install in them.

[tj]

yeah that would be great, if it makes sense to isaac, I might be missing something

[Mithgol]

Running npm install on dependencies would probably fail in a somewhat important use case where a bundled dependency contains a pre-compiled module so that npm install of parent module _just works_ (and does not require Python and Visual Studio on Windows to build that one dependency).

In any case a bundled dependency is probably bundled for the very reason of avoiding simple npm install — otherwise the author of the parent module would make it unbundled (and would let npm install it).

In your own use case you should have not only bundled your unpublished modules to the parent module's npm package — that's not enough; you should also have copied and added children's dependencies to the parent module's package.json. (I believe you may accomplish it unless their dependencies are conflicting.)

Then npm install will install those dependencies (to the parent's node_modules folder instead of children's), and then Node.js will scan parent's node_modules folder immediately after children's folders — and there it'll find all the dependencies necessary for its children.

Problem solved.

[aseemk]

Definitely +1 to this issue. FYI that this is probably covered through a combination of #1341 and #1558.

@Mithgol:

[…] you should also have copied and added children's dependencies to the parent module's package.json. (I believe you may accomplish it unless their dependencies are conflicting.)

I don't believe this is a recommended practice. It goes against the whole purpose of modularity -- you'll have to keep your own package.json in sync with the module's. And indeed, this won't support conflicting dependencies, which was the reasoning behind npm 1.0's move to local node_modules.

[tj]

yeah.. that wont really help me here, that's like going back to git submodules

[tj]

@isaacs this is the thing I was talking about at the conf. I'll still try the git repo route but it's a little limiting since we're using github to host the repos and we're already at our private repo limit

[aseemk]

Not sure what the git repo route is exactly, but if it's specifying your deps as git URLs, we tried that too and found that a bit inflexible to use -- e.g. in staging, you want to use the staging branch while in production you want to use the production branch.

[tj]

the biggest problem for us with the git repo thing is just the amount of private repos we would need. The solution @mmalecki mentioned sounds reasonable to me but I know isaac mentioned those are strictly untouched by npm by design so maybe we need something else

[isaacs]

npm should definitely do this.

At the moment, bundledDependencies are just ignored. It would be best if npm skipped the unpack step, but still traversed into the child folders to see what needs to be done.

It wouldn't be terribly hard to do.

[tj]

haha this is our clean install time since it has to loop each dep:

real    3m45.342s
user    1m14.221s
sys 0m14.632s

[tj]

with all the duplicates we have over 12,000 require()s now and booting takes about 5s hahaha :D

[isaacs]

@visionmedia You should try running npm dedupe on your tree. Should make your startup time a bit faster.

[tj]

k cool I will take a look at that

[therealplato]

+1

[jerryorr]

+1

[mkuklis]

+1

[isaacs]

+1

[tommedema]

+1

[tglines]

+1

[justinwalsh]

+1

[icodeforlove]

+1

[andrewluetgers]

+1

[isaacs]

+1

[mercmobily]

+1. I just got bitten by this one too, and I just couldn't figure out why dependencies weren't getting installed...!

[sidwood]

+1

[vjpr]

+1

[isaacs]

+1

[robertjd]

+1

[robertkowalski]

ok, just to make sure I am getting this right:

npm install should traverse the bundledDependencies-Array and based on that entries it then tries to enter the folders of this entries and does a npm install inside them?

[Kingdutch]

@robertkowalski That seems to be the desired behaviour yes. (It's also what I expected when I added a bundledDependency and gave it a package.json like a good boy scout would and ran npm install on my base module)

[sethkinast]

Just got bitten by this as well, +1

[mercmobily]

I am writing a module that has twenty-five (!) bundleDependencies, each one with different dependencies. At the moment, I am making sure that all dependencies of each bundled module are added to the main package.js file of the main project. But... it's ugly. So, I confirm:

npm install should traverse the bundledDependencies-Array and based on that entries it then tries to enter the folders of this entries and does a npm install inside them"

Yes, absolutely, please.

[mikermcneil]

Currently, we're doing this for copying deps (which looks for subdeps and copies them, similar to @mercmobily's solution).

I've been struggling with the best way to do this as well-- was thinking maybe a script that makes a one-time-use package.json for the bundleds+the normal dependencies from the main module's package json? Temporarily replaces the existing package.json file, npm installs, then writes the original package.json back in place when it's done. That would allow us to take advantage of the dependency deduping already built into npm, and not have to worry about keeping anything rerolled in sync.

In case this helps anyone, here's a similar script that looks at a testDependencies key in the package.json file and npm installs them (in case you have a situation where you don't want devDependencies to bring in everything at once).

[Philzen]

bump (Ouch!) 👍

[mercmobily]

@mikermcneil Ugh I wish I had used a script as well. Right now I did the "checking" work manually, and live with the thought that when/if this bug is fixed, I will have to go through everything and take out the redundant dependences from my main package...

[thomasfr]

+1 :D

[mixxen]

+1

[tjwebb]

was this ever resolved? I could be misunderstanding, but it seems because of this issue the behavior of bundledDependencies doesn't align with the documentation

[nkbt]

I don't think this will be ever (or soon) resolved. See above. I personally cannot see any automated solution to prevent indefinite recursive libs fetching. What could be done - some additional config variable with list of dependencies that must be fetched even if they exist in parents.

Just in case here is my recent talk on the topic http://nkbt.github.io/nnj with failing example and couple of ways to solve the issue.

[thomasfr]

Thanks @nkbt for your link!
I recently also tried some variations to organize local libs / modules whatever one might call it. I tried for instance some variations with setting NODE_PATH. I also tried of adding a node_modules/lib dir so i could do require("lib/dothisandalsothat.js (without relative file paths.). It all worked but it all felt a bit cumbersome or not right if we have npm at our hands.

How do others manage its local private node.js modules and libraries?
cheers

[othiym23]

We recently landed a new feature that allows you to npm install --save / depend on file paths (i.e. "foo" : "file:../../foo-local"), which is basically what this issue is requesting. Give that a shot, and let's iterate from there. It took us a while, but we eventually we got around to this. Thanks to everyone for the ideas and suggestions.

[mercmobily]

Hi,

The documentation says:

"This feature is helpful for local offline development and creating tests
that require npm installing where you don't want to hit an external server,
but should not be used when publishing packages to the public registry."

That's not quite my use case. In my use case, I have Hotplate which in turn
contains, in stock, a number of modules (36 of them to be exact), each
one with a specific task and -- potentially -- with different requirements
in terms of sub-modules.

I am not sure what you are saying actually closes this ticket...?

Merc.

On 16 September 2014 11:58, Forrest L Norvell notifications@github.com
wrote:

We recently landed
https://github.com/npm/npm/blob/b706d637d5965dbf8f7ce07dc5c4bc80887f30d8/doc/files/package.json.md#local-paths
a new feature that allows you to npm install --save / depend on file
paths (i.e. "foo" : "file:../../foo-local"), which is basically what this
issue is requesting. Give that a shot, and let's iterate from there. It
took us a while, but we eventually we got around to this. Thanks to
everyone for the ideas and suggestions.

—
Reply to this email directly or view it on GitHub
#2442 (comment).

[thomasfr]

@othiym23 Nice addition. I will give that a try - looks good! :)

[nathanbowser]

@othiym23 This is so amazing. Thank you!

[othiym23]

@mercmobily

There are two parts (at least) to this issue -- one is being able to effectively us local dependencies (which is what the issue is titled after), and another is ensuring that dependencies (including bundledDependencies) are fully installed after npm install runs. We've got the local dependencies piece of it covered, even if it doesn't work exactly the same way as described by @visionmedia / using bundledDependencies. The rest is something that can be addressed by the multi-stage install work that @iarna is doing, so I'll draw her attention to this discussion and we'll make sure that bundledDependencies are part of the scope of her work. There are many, many issues open around this, so because the biggest piece of this use case has been addressed, I'm leaving it closed (for now).

[mercmobily]

Thanks Forrest. Which one is the ticket that @iarna is working on, about the multi-stage install? I
just want to make sure my use case is at least considered!

[othiym23]

@mercmobily

There isn't a specific tracking issue for your particular use case, so can you open a new issue describing it (as concretely as you're able)? The multi-stage install project is meant to rework the installation process so that it figures out exactly which dependencies are missing, and then installing all of them in a single pass. It seems entirely in scope for bundledDependencies to be included as part of that dependency tree realization, but an explicit use case (or maybe even… a failing test 😍) would be very useful.

[mercmobily]

Just wrote out a ticket, and then realised that it already existed...!

#1341

And that the multi-stage install should really fix it:

#5919

So, all good -- thanks!