Any way to separate production from dev dependencies in node_modules in npm 3?

[mhart]

We currently check in our production dependencies to git (which allows us to git checkout a full working version of our app on our production machines), and we don't check in our dev dependencies.

Until npm@3 we've been able to specify our dev dependency directories in our .gitignore:

node_modules/**/.bin
node_modules/browserify
node_modules/watchify

There's a duplication here with what's listed in package.json and ideally there'd be a single line we could ignore (eg node_modules_dev or something), but up until npm@3 it wasn't too bad.

The issue under npm@3 is we have literally hundreds of packages at the top level that need to be ignored – and worse, I worry that we may ignore some only to have them needed for a production dependency down the track.

node_modules/**/.bin
node_modules/browserify
node_modules/watchify
node_modules/acorn
node_modules/ansi-green
node_modules/ansi-wrap
node_modules/anymatch
node_modules/arr-diff
node_modules/arr-flatten
node_modules/array-filter
node_modules/array-map
node_modules/array-reduce
node_modules/array-slice
node_modules/array-unique
node_modules/arrify
node_modules/asn1.js
node_modules/assert
node_modules/astw
node_modules/async-each
node_modules/balanced-match
node_modules/base64-js
node_modules/binary-extensions
node_modules/bn.js
node_modules/brace-expansion
node_modules/braces
node_modules/brorand
node_modules/browser-pack
node_modules/browser-resolve
node_modules/browserify-aes
node_modules/browserify-rsa
node_modules/browserify-sign
node_modules/browserify-transform-tools
node_modules/browserify-zlib
node_modules/buffer-xor
node_modules/buffer
node_modules/builtin-status-codes
node_modules/builtins

Etc...

Is there anything we can do about this? Any npm command that I might be missing that will help us out? Any way to separate production from dev dependencies?

[mhart]

Just realised we're going to run into the same issue with our .dockerignore files – so even if we stop checking in deps to git and move to npm shrinkwrap, our Docker images are still going to explode in size because we can't specify which directories to ignore when building (well, not easily anyway)

[mhart]

(unless we ignore the entire node_modules dir and do the npm install as a step in our Dockerfile, which I was kinda hoping to avoid as our base Docker image doesn't have npm on it)

[bcomnes]

Would the --production flag come in handy here? You can choose to not install dev deps that way.

[mhart]

@bcomnes right, but we check our production deps into git – which means during development (when we just do a regular npm install) we have many, many deps that aren't checked in to git – and that we don't want checked in to git, so we want them ignored.

[ashleygwilliams]

@mhart generally, i ignore the entire node_modules folder and npm install as part of deploy. is there a reason you don't want to do this? adding npm to the docker image isn't the worst.

[mhart]

@ashleygwilliams – yeah, a number of reasons. Even though this is a bit too strongly worded, I agree with a bunch of it: https://gist.github.com/sukima/3854887#doesnt-checking-in-node_modules-create-a-lot-of-noise-in-the-source-tree-that-isnt-related-to-my-app

Also, even though npm has gotten more reliable over the years, reducing deployment dependencies on it has its advantages. Especially if you have your git repo accessible internally, then you're not relying on any external services to be up and fully functioning when deploying. Plus it's sooooo much faster than npm installing.

We use shrinkwrap on some projects, so I'm not totally averse to moving over our other projects and just having to rely on npm, but I'd like to explore whether it's possible to avoid first.

Same with the docker issue, yeah, it can be done and it'll increase our Docker build time, etc, etc, but then we're bending to npm's will – I'm trying to explore whether there's ways we don't have to do that.

Being able to divide between production and development modules seems to be something that will pop up in a number of circumstances – like, I dunno, let's say I just want to delete my dev deps – how do I do that? Maybe there's a command to help with that already? If there is, then I could use that!

[mhart]

(reading that blog post again makes me chuckle, I wonder if @mikeal still feels the same way... "All you people who added node_modules to your gitignore, remove that shit, today, it’s an artifact of an era we’re all too happy to leave behind." 😸 )

[ashleygwilliams]

@mhart, i think you are right that splitting between dev and prod modules is something that will come up again.

i'm looking at bundleddependencies right now, there might be a way to use that to finagle a solution?

looks like you could autogenerate them with this: https://www.npmjs.com/package/bundled-dependencies

[mikeal]

The era of global modules is dead.

This is definitely still true.

The rest I'm not so sure about, I haven't had enough time to mess with npm3 to be able to tell. If you don't have native module dependencies I still think that checking them into git is easier and less error prone than other solutions, but it gets nasty with native modules because people checkin build artifacts which then cause git conflicts down the road on production machines. It's hard to call any solution which inevitably requires you to ssh in to production machines and manually resolve conflicts "simple."

[mhart]

@ashleygwilliams so something like that, and then npm pack to get a tarball? So you end up with the same thing as npm install --production. Doesn't solve the git issue, but we could use that to send to Docker. Nice idea! Just hope none of these come into play... https://github.com/npm/npm/search?q=bundledDependencies&state=open&type=Issues&utf8=%E2%9C%93

I've also just discovered npm ls --prod --parseable and npm ls --dev --parseable which look like they may be able to make this a bit easier – I just need to wrangle them and a little and see if there's any overlap between the two 😄

[mhart]

Hmm, there is overlap unfortunately (eg, node_modules/readable-stream appears in both --prod and --dev output) – so I'll need to do some diffing/subtracting of some sort

[mhart]

@mikeal native modules seem to be ok – but unless you're happy checking in all your dev deps as well as your production deps, then you might not be a happy camper with npm 3 + git at this stage.

[othiym23]

What about running npm prune --prod before updating node_modules in git? (As others have said, npm install --only=prod will keep dev modules from being installed in the first place.) Using one of those two commands should give you the behavior you're looking for.

[mhart]

@othiym23 during development it's not just the act of updating node_modules though – it's any git commit, any git diff, any git command at all. Basically we'd have to run npm prune --prod before any git command so that our output isn't messed with by hundreds of lines of dev deps. And then npm install again to get them all back again...

[mhart]

@othiym23 but that certainly does answer the question of how you delete dev deps, thanks!

[mhart]

So... this command will at least give me a list of dev deps that don't overlap with prod deps... I think:

diff -U 10000 <(npm ls --dev --parseable | sort) <(npm ls --prod --parseable | sort) | grep '^-' | sed 's/^-//g'

So I can copy/paste the output of that (with some modifications to get rid of sub-deps) into my .gitignore and .dockerignore

Seems... hmmm, ugly though (cause that's still a 400 line gitignore...)

[othiym23]

This is now on the road map (hence the dependencies label), and this also bites us too, which is a good sign that it will get addressed in the non-too-distant future.

[mhart]

Excellent news! 👍

[ashleygwilliams]

nice! 📦 🎉

[samartioli]

I don't know if this is the write place to put this so please let me know if not. But along with separating dev and prod dependencies, npm should also separate primary and secondary dependencies. It is a cluster now to open up node_modules on any sizable project. way too many folders!
This would be ideal:

node_modules/
    dev
        [primary modules (only those listed in package.json]
        secondary/
            [secondary modules]
    prod
        [primary modules (only those listed in package.json]
        secondary/
            [secondary modules]

cc @johnpapa

[othiym23]

After a bit of discussion, the CLI team has an idea of how this might be made to work. This is a candidate implementation which only requires a few small changes to npm that would also be useful in other contexts:

1. Add two new switches to npm ls, --physical, and --logical. npm ls --logical gets you what you get today, printing out the logical tree of dependencies. --physical would be a way for npm to print out the tree as it's laid out on disk without forcing the use of external tooling, and would also allow you to use the npm ls <package> command npm has today, to show all instances of <package> in the physical tree.
2. Extend the semantics of --only={dev,prod} and --also={dev,prod} to npm ls such that they show only those packages that are needed for production or development deploys. This would also probably require us to tweak the use of --production and --development and document the whole thing very carefully so it doesn't prove to be more confusing than useful.
3. Users who check in node_modules write a package script thusly: "update-gitignore": "npm ls --physical --depth=0 --parseable --only=dev > node_modules/.gitignore". This gets the list of dependencies that are only used by devDependencies (which rules out the shared dependencies of regular dependencies mentioned above) and sticks it in the specific .gitignore it needs to be in.

There are a lot of (relatively small) tweaks that need to happen here to get this landed, but they're all things that would be useful on their own. We'll work on getting to this in the next year, because this is an important workflow to support, but if people want to tackle some or all aspects of the problem above, patches are very very welcome (very).

[xealot]

I know that this is being taken into consideration, which is great. But I haven't seen anyone mention private, corporate and enterprise NPM as reasons for needing this. We have a corporate NPM account and have to ship an absolute ton of JS up to Docker because we have no good way to send only what we need. We're unwilling to script an npm login to run npm install.

Hence, even if we wanted to run npm install remotely, which we really don't. We can't without logging in on the container.

[gonzofish]

to tack on to what @xealot said, we have a private NPM so that we can have a central repo for common components.

Our company delivers code to customers who then build the code on their systems; the issue is that they won't be able to install those common components since they only reside in our private NPM.

The way we've gotten around it up until now is to use Bower for the common components, but I think that's going the way of the dinosaur and would love to use NPM for everything.

[ORESoftware]

anybody happen to know if there is a difference between:

npm install --production

and

npm install --only=production

?

[thomshouse]

Just FYI, another approach one could take would be to write your node_modules/.gitignore file to ignore all by default, but negate production dependencies.

This results in a file with the following structure:

/*
/.gitignore
!/production_package_example
...

I think this is a simple and effective approach, if a bit inelegant. Also, it should result in a smaller .gitignore for people who use node modules mostly for development purposes.

[mhart]

@thomshouse that works with npm@2, but the problem is, in npm@3 with a flattened hierarchy, you don't know what to ignore and what not to.

How do you know that a particular package is being used by a prod dep?

[thomshouse]

@mhart I'm on npm 3.5.3 and it appears to me that npm ls --prod --parseable appears to return production dependencies for me. Am I missing something?

[mhart]

@thomshouse er, I think I'm missing something – your comment was about a .gitignore file

[mhart]

@thomshouse sorry, I thought in your original comment you were suggesting to just manually add your top-level prod deps in .gitignore – are you actually talking about a command-based approach similar to #9674 (comment) ?

[thomshouse]

@mhart Correct! Sorry, I realize now that my suggestion was slightly off topic. Still, hopefully it's at least somewhat relevant to what people are trying to accomplish.

And yes, I was suggesting a command-based approach. I hesitated to share the command I was using as it was somewhat of a quick hack, but here it is:

cd node_modules 2> /dev/null && cd .. && echo '/*' > node_modules/.gitignore && echo '!/.gitignore' >> node_modules/.gitignore && npm ls --prod --parseable 2> /dev/null | grep node_modules | sed 's/.*node_modules/!/' >> node_modules/.gitignore

It expects to be run from a path containing a package.json file, i.e. the same directory containing the node_modules directory. I believe the way I'm running the npm ls results through sed should allow an npm@2 user to build a .gitignore file that will be forward-compatible with npm@3, but I have not tested this.

[legodude17]

@ORESoftware I am pretty sure that --production is deprecated.

[sonicdoe]

@legodude17 Do you have a citation for --production being deprecated? I couldn’t find anything in npm-ls’s docs or npm’s changelogs.

According to the documentation, --only=production is just an alias to --production.

[legodude17]

@sonicdoe thanks for pointing that out! I was wrong about that. In that case, @ORESoftware they are the same.

[dpolivy]

Since there are no commits linked to this issue, it doesn't seem like it's had much traction in the past year and a half. I'm migrating to npm3 and running into this workflow issue myself. What's the 'best' current alternative to either filter out dev dependencies or explicitly include production dependencies in an npm3 flat hierarchy? I'm on Windows (so tools/commands may need to be tweaked).

[sonicdoe]

You could take a look at the --global-style argument. From the documentation:

The --global-style argument will cause npm to install the package into your local node_modules folder with the same layout it uses with the global node_modules folder. Only your direct dependencies will show in node_modules and everything they depend on will be flattened in their node_modules folders. This obviously will eliminate some deduping.

While this doesn’t allow you to exclude development dependencies in a single line, it at least allows you to exclude them by listing the direct dependencies in .gitignore (as shown in @mhart’s original post).