Best way to set --max-old-space-size when running npm?

[codeimpossible]

We have some pretty large builds at Hudl, and we've been seeing npm install fail with ECONNRESET errors. We spent a lot of time debugging the issue and are fairly confident the failures were due to npm hitting the upper limit of memory within node and being terminated prematurely.

We changed our npm.cmd file to set --max-old-space-size to 4gb when starting npm, and this fixed our issue, but we're wondering if there is a better way to set this parameter when running npm?

[othiym23]

There's not a really good way to pass arbitrary parameters to the Node process from the command line; maybe there should be. Pointing this at the team for further discussion in product triage.

[codeimpossible]

Cool, thanks @othiym23! Also, have you heard of other teams running up against memory limits when installing npm dependencies? Any tips/tricks to reduce our memory footprint? We're going to be going through our packages and trimming out anything that we don't need, but just curious as to how common or uncommon this is.

[othiym23]

It's not super common that this needs to be done, mostly (I think) because most of the time people are doing deploys and installs from fairly beefy vms with the gig or so of RAM that npm needs to run efficiently. When it's an issue, it's generally on Raspberry Pis and other small machines, but it's fairly rare for installs even then – this is mostly an issue with using npm's not-too-beloved search. Also, I'm a little confused by the correlation you've drawn between ECONNRESET and out of memory – typically when a Node process hits memory limits, it just crashes, not drops connections. Is this on a local cache or something?

Either way, being able to pass parameters to the runtime like you suggest is a super handy, if rarely necessary, tool, and it would be nice to have in npm. Unfortunately, Node itself doesn't allow configuration via environment variables, and npm on non-Windows systems is run directly as a Node script, so we'd have to write a new shell script to handle this, which would require some coördination with the Node.js team that's responsible for bundling and installing npm. This is something the npm CLI team is going to try to get to in the next year or so, but if somebody else wants to take a stab at it first, that would be great. Thanks!

[codeimpossible]

@othiym23 thanks for the writeup, the correlation is definitely odd but something we noticed when running installs against our in-house npm server (sinopia). When the memory would grow beyond 1.7gb the node process would exit and we would see ECONNRESET errors in our build logs. Increasing the limit did seem to make installs more reliable, but this was likely just a coincidence.

After more debugging today we're pretty sure this is some sort of networking issue between our internal npm server (sinopia) and our build agents, since like you said the process should just exit immediately if we were overflowing the memory limit. And we've had repeated incidents of ECONNRESET with the newly up'd memory limits.

Thanks again for the write-up and adding the feature to your backlog - the work you and the rest the cli team do is greatly appreciated!

[daavve]

I use aliasing to pass options to node for npm. Here is my .bashrc:

alias npm='node --max_old_space_size=8000 /usr/bin/npm'

[MichaelJCole]

@othiym23 I don't think this is an edge use-case. Do I need to hard-code this in my docker image, even though the memory is allocated at runtime? Is there a reason this isn't an environment variable?

[ebdrup]

We nee this quite often, when running large things using npm scripts. npm run XXX. Would love to see a fix for this.

[bchr02]

@othiym23 so is it max_old_space_size or max-old-space-size (dashes or underscores)?

[bchr02]

nevermind, I found it by RTFM here:

--v8-options

Added in: v0.1.3
Print v8 command line options.

Note: v8 options allow words to be separated by both dashes (-) or underscores (_).

For example, --stack-trace-limit is equivalent to --stack_trace_limit.

[maxgalbu]

There IS an easy way to pass parameter down to node from npm:

npm run something -- --max-old-space-size=250 --anything-you-want

[kenany]

@maxgalbu As far as I know, that just passes those args down to whatever your "something" script is.

[rmorlang]

I had success with:

node --max-old-space-size=250 `which npm` some_npm_command

[majorcool]

I faced this issue too, and found this npm package helpful increase-memory-limit

[nodkrot]

Example to previous answer

node --max_old_space_size=2000 `which npm` start

[chamini2]

I would have expected to be able to pass node parameters with npm. Something like (for example):

$ npm run start --node-flags --max-old-space-size=512 --no-warnings

[nikhilo]

@MichaelJCole Were you able to find out a solution for setting the value in Docker container (at run time) ?

[sandorfr]

The package suggested by @EXin it a good workaround for now.

I have the feeling this is mostly a problem with webpack or rollup and they might be running on a child process of the actual process started with the npm run command. For that reason, it might be there responsibility to tweak their commands so people don't run into those issues.

[chamini2]

I have the feeling this is mostly a problem with webpack or rollup and they might be running on a child process of the actual process started with the npm run command. For that reason, it might be there responsibility to tweak their commands so people don't run into those issues.

I disagree, maybe the tweaking is necessary too, but I should be able to pass these flags to node.

[HipsterZipster]

It'd be even nicer if there was a ENV_VAR to set this for an entire system

[lostpebble]

@HipsterZipster it's not perfect, but you could set it up for yourself. I'm currently doing this, which seems to work well:

scripts: {
    start: "cross-var node --max-old-space-size=$NODE_JS_MAX_OLD_SPACE_SIZE --max-semi-space-size=$NODE_JS_MAX_SEMI_SPACE_SIZE ./server.js",
  },

cross-var is a module which allows you to use environment variables cross-platform in the way I'm using them there.

[kuncevic]

any news if that is coming?
How would you make this node --max_old_space_size=2000 which npm start to work on windows, seems like which = where on windows but wont work replacing just that.

There is one more bash solution https://stackoverflow.com/a/40939496/415078 wonder is there is a way to do something similar on Windows:
node --max_old_space_size=8000 /usr/bin/npm install -g ionic

[mischkl]

desperately needed here too! cross-platform team running into issues with webpack, especially on windows. would be great to be able to specify max-old-space-size via the npm script

[mischkl]

So it turns out that instead of needing to alias npm or otherwise call node directly, you can increase Node's max heap size by setting the NODE_OPTIONS environmental variable (introduced in Node 8) as follows:

NODE_OPTIONS=--max_old_space_size=4096

Usage with NPM scripts:

"scripts": {
    "start": "cross-env NODE_OPTIONS=--max_old_space_size=4096 webpack"
}

Note that it's important to specify the option with_underscores since that's the only one that NODE_OPTIONS accepts.

[noumaans]

NODE_OPTIONS=--max-old-space-size=4096
Please use hyphens instead of underscores if setting it as an environment variable in your shell or through Makefile.

https://nodejs.org/dist/latest-v8.x/docs/api/cli.html#cli_node_options_options

[mischkl]

It turns out either way should work, see nodejs/node#14093. I didn't see that documentation and was basing my statement on the source code, but I overlooked that either variant is allowed.

[AnyhowStep]

Is there a way to set the --max-old-space-size to more than 8GB?
I'm trying to type check a TypeScript project but I keep running into out of memory errors.

I tried setting it to 16GB but it still just uses 8GB before giving up.

[EDIT]
At this point, I just want to see how much RAM I can throw at tsc till it compiles. If possible, I'd allocate all 32 GB and then throw in swap space as well.

[huan]

@AnyhowStep I'm interested in how to increase the memory to 32GB too.

Did you get any progress about this?

BTW: it seems that you have a huge TypeScript project!

[AnyhowStep]

@zixia

It's not so much huge as it is complicated, I think.

I never did figure it out, unfortunately.

[flash-me]

1. Proposal

You can create a .npmrc in your project (see also project-config-file) and define your options there:
NODE_OPTIONS=--max-old-space-size=4096
npm config ls to see the output.

2. Proposal

You add a config field in your package.json and define your options there.

I'll try out both of this myself. I hope they work, since these would a much cleaner approach instead of writing "node ..." in front of every npm script...
And to ensure that everyone has the same project settings without touching userenvironment config.

Edit

Both does not work. I couldn't find any solution to set this value within project configurations. There is neither the possibility to run a wildcard script before any npm script. Either you have to set the user environment variable (which you can't access for good reasons within npm) or you have to insert it in all npm scripts, which is quite ugly...