local private module dependencies #2442

Closed
tj opened this Issue May 10, 2012 · 49 comments
@tj
tj commented May 10, 2012

hey hey, looking for your thoughts or suggestions on this issue. What I wanted ideally is to modularize our app(s) using vanilla node_modules, however they are not in any registry so they're checked into the repo. $ npm install see's that they're already installed and ignores them, even if their own dependencies might not be installed. I have bigger plans for this so ideally avoiding a shell script to iterate and $ cd node_modules/pkg && npm install if possible.

Any suggestions for this use-case? maybe if "private": true is specified (or similar) npm could check the deps.

@mmalecki
npm member

I think npm should just iterate over bundledDependencies and run npm install in them.

@tj
tj commented May 10, 2012

yeah that would be great, if it makes sense to isaac, I might be missing something

@Mithgol

Running npm install on dependencies would probably fail in a somewhat important use case where a bundled dependency contains a pre-compiled module so that npm install of parent module just works (and does not require Python and Visual Studio on Windows to build that one dependency).

In any case a bundled dependency is probably bundled for the very reason of avoiding simple npm install — otherwise the author of the parent module would make it unbundled (and would let npm install it).

In your own use case you should have not only bundled your unpublished modules to the parent module's npm package — that's not enough; you should also have copied and added children's dependencies to the parent module's package.json. (I believe you may accomplish it unless their dependencies are conflicting.)

Then npm install will install those dependencies (to the parent's node_modules folder instead of children's), and then Node.js will scan parent's node_modules folder immediately after children's folders — and there it'll find all the dependencies necessary for its children.

Problem solved.

@aseemk

Definitely +1 to this issue. FYI that this is probably covered through a combination of #1341 and #1558.

@Mithgol:

[…] you should also have copied and added children's dependencies to the parent module's package.json. (I believe you may accomplish it unless their dependencies are conflicting.)

I don't believe this is a recommended practice. It goes against the whole purpose of modularity -- you'll have to keep your own package.json in sync with the module's. And indeed, this won't support conflicting dependencies, which was the reasoning behind npm 1.0's move to local node_modules.

@tj
tj commented May 11, 2012

yeah.. that wont really help me here, that's like going back to git submodules

@tj
tj commented May 21, 2012

@isaacs this is the thing I was talking about at the conf. I'll still try the git repo route but it's a little limiting since we're using github to host the repos and we're already at our private repo limit

@aseemk

Not sure what the git repo route is exactly, but if it's specifying your deps as git URLs, we tried that too and found that a bit inflexible to use -- e.g. in staging, you want to use the staging branch while in production you want to use the production branch.

@tj
tj commented May 28, 2012

the biggest problem for us with the git repo thing is just the amount of private repos we would need. The solution @mmalecki mentioned sounds reasonable to me but I know isaac mentioned those are strictly untouched by npm by design so maybe we need something else

@isaacs
npm member

npm should definitely do this.

At the moment, bundledDependencies are just ignored. It would be best if npm skipped the unpack step, but still traversed into the child folders to see what needs to be done.

It wouldn't be terribly hard to do.

@tj
tj commented Jul 23, 2012

haha this is our clean install time since it has to loop each dep:

real    3m45.342s
user    1m14.221s
sys 0m14.632s
@tj
tj commented Aug 14, 2012

with all the duplicates we have over 12,000 require()s now and booting takes about 5s hahaha :D

@isaacs
npm member

@visionmedia You should try running npm dedupe on your tree. Should make your startup time a bit faster.

@tj
tj commented Aug 22, 2012

k cool I will take a look at that

@mkuklis

+1

@isaacs
npm member

+1

@tglines

+1

@isaacs
npm member

+1

@mercmobily

+1. I just got bitten by this one too, and I just couldn't figure out why dependencies weren't getting installed...!

@sidwood

+1

@vjpr

+1

@isaacs
npm member

+1

@robertjd

+1

@robertkowalski
npm member

ok, just to make sure I am getting this right:

npm install should traverse the bundledDependencies-Array and based on that entries it then tries to enter the folders of this entries and does a npm install inside them?

@Kingdutch

@robertkowalski That seems to be the desired behaviour yes. (It's also what I expected when I added a bundledDependency and gave it a package.json like a good boy scout would and ran npm install on my base module)

@hurrymaplelad hurrymaplelad referenced this issue in yeoman/generator Sep 1, 2013
Closed

Project specific generators #348

@sethkinast

Just got bitten by this as well, +1

@mercmobily

I am writing a module that has twenty-five (!) bundleDependencies, each one with different dependencies. At the moment, I am making sure that all dependencies of each bundled module are added to the main package.js file of the main project. But... it's ugly. So, I confirm:

npm install should traverse the bundledDependencies-Array and based on that entries it then tries to enter the folders of this entries and does a npm install inside them"

Yes, absolutely, please.

@mikermcneil

Currently, we're doing this for copying deps (which looks for subdeps and copies them, similar to @mercmobily's solution).

I've been struggling with the best way to do this as well-- was thinking maybe a script that makes a one-time-use package.json for the bundleds+the normal dependencies from the main module's package json? Temporarily replaces the existing package.json file, npm installs, then writes the original package.json back in place when it's done. That would allow us to take advantage of the dependency deduping already built into npm, and not have to worry about keeping anything rerolled in sync.

In case this helps anyone, here's a similar script that looks at a testDependencies key in the package.json file and npm installs them (in case you have a situation where you don't want devDependencies to bring in everything at once).

@MarcDiethelm MarcDiethelm added a commit to MarcDiethelm/xtc-cli that referenced this issue Apr 2, 2014
@MarcDiethelm MarcDiethelm bundledDependencies don't have their dependencies installed: npm/npm#…
…2442 need to do that ourselves
ee04c92
@Philzen

*bump* (Ouch!) 👍

@mercmobily

@mikermcneil Ugh I wish I had used a script as well. Right now I did the "checking" work manually, and live with the thought that when/if this bug is fixed, I will have to go through everything and take out the redundant dependences from my main package...

@thomasfr

+1 :D

@mixxen

+1

@tjwebb

was this ever resolved? I could be misunderstanding, but it seems because of this issue the behavior of bundledDependencies doesn't align with the documentation

@nkbt

I don't think this will be ever (or soon) resolved. See above. I personally cannot see any automated solution to prevent indefinite recursive libs fetching. What could be done - some additional config variable with list of dependencies that must be fetched even if they exist in parents.

Just in case here is my recent talk on the topic http://nkbt.github.io/nnj with failing example and couple of ways to solve the issue.

@thomasfr

Thanks @nkbt for your link!
I recently also tried some variations to organize local libs / modules whatever one might call it. I tried for instance some variations with setting NODE_PATH. I also tried of adding a node_modules/lib dir so i could do require("lib/dothisandalsothat.js (without relative file paths.). It all worked but it all felt a bit cumbersome or not right if we have npm at our hands.

How do others manage its local private node.js modules and libraries?
cheers

@othiym23

We recently landed a new feature that allows you to npm install --save / depend on file paths (i.e. "foo" : "file:../../foo-local"), which is basically what this issue is requesting. Give that a shot, and let's iterate from there. It took us a while, but we eventually we got around to this. Thanks to everyone for the ideas and suggestions.

@othiym23 othiym23 closed this Sep 16, 2014
@mercmobily
@thomasfr

@othiym23 Nice addition. I will give that a try - looks good! :)

@nathanbowser

@othiym23 This is so amazing. Thank you!

@othiym23

@mercmobily

There are two parts (at least) to this issue -- one is being able to effectively us local dependencies (which is what the issue is titled after), and another is ensuring that dependencies (including bundledDependencies) are fully installed after npm install runs. We've got the local dependencies piece of it covered, even if it doesn't work exactly the same way as described by @visionmedia / using bundledDependencies. The rest is something that can be addressed by the multi-stage install work that @iarna is doing, so I'll draw her attention to this discussion and we'll make sure that bundledDependencies are part of the scope of her work. There are many, many issues open around this, so because the biggest piece of this use case has been addressed, I'm leaving it closed (for now).

@mercmobily
@othiym23 othiym23 added this to the multi-stage install milestone Sep 16, 2014
@othiym23

@mercmobily

There isn't a specific tracking issue for your particular use case, so can you open a new issue describing it (as concretely as you're able)? The multi-stage install project is meant to rework the installation process so that it figures out exactly which dependencies are missing, and then installing all of them in a single pass. It seems entirely in scope for bundledDependencies to be included as part of that dependency tree realization, but an explicit use case (or maybe even… a failing test 😍) would be very useful.

@mercmobily

Just wrote out a ticket, and then realised that it already existed...!

#1341

And that the multi-stage install should really fix it:

#5919

So, all good -- thanks!

@iarna iarna locked and limited conversation to collaborators Jun 24, 2015
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.