Skip to content
This repository has been archived by the owner. It is now read-only.

Best way to set --max-old-space-size when running npm? #12238

Open
codeimpossible opened this issue Apr 6, 2016 · 31 comments
Open

Best way to set --max-old-space-size when running npm? #12238

codeimpossible opened this issue Apr 6, 2016 · 31 comments

Comments

@codeimpossible
Copy link

@codeimpossible codeimpossible commented Apr 6, 2016

We have some pretty large builds at Hudl, and we've been seeing npm install fail with ECONNRESET errors. We spent a lot of time debugging the issue and are fairly confident the failures were due to npm hitting the upper limit of memory within node and being terminated prematurely.

We changed our npm.cmd file to set --max-old-space-size to 4gb when starting npm, and this fixed our issue, but we're wondering if there is a better way to set this parameter when running npm?

@othiym23
Copy link
Contributor

@othiym23 othiym23 commented Apr 7, 2016

There's not a really good way to pass arbitrary parameters to the Node process from the command line; maybe there should be. Pointing this at the team for further discussion in product triage.

@codeimpossible
Copy link
Author

@codeimpossible codeimpossible commented Apr 7, 2016

Cool, thanks @othiym23! Also, have you heard of other teams running up against memory limits when installing npm dependencies? Any tips/tricks to reduce our memory footprint? We're going to be going through our packages and trimming out anything that we don't need, but just curious as to how common or uncommon this is.

@othiym23
Copy link
Contributor

@othiym23 othiym23 commented Apr 7, 2016

It's not super common that this needs to be done, mostly (I think) because most of the time people are doing deploys and installs from fairly beefy vms with the gig or so of RAM that npm needs to run efficiently. When it's an issue, it's generally on Raspberry Pis and other small machines, but it's fairly rare for installs even then – this is mostly an issue with using npm's not-too-beloved search. Also, I'm a little confused by the correlation you've drawn between ECONNRESET and out of memory – typically when a Node process hits memory limits, it just crashes, not drops connections. Is this on a local cache or something?

Either way, being able to pass parameters to the runtime like you suggest is a super handy, if rarely necessary, tool, and it would be nice to have in npm. Unfortunately, Node itself doesn't allow configuration via environment variables, and npm on non-Windows systems is run directly as a Node script, so we'd have to write a new shell script to handle this, which would require some coördination with the Node.js team that's responsible for bundling and installing npm. This is something the npm CLI team is going to try to get to in the next year or so, but if somebody else wants to take a stab at it first, that would be great. Thanks!

@codeimpossible
Copy link
Author

@codeimpossible codeimpossible commented Apr 7, 2016

@othiym23 thanks for the writeup, the correlation is definitely odd but something we noticed when running installs against our in-house npm server (sinopia). When the memory would grow beyond 1.7gb the node process would exit and we would see ECONNRESET errors in our build logs. Increasing the limit did seem to make installs more reliable, but this was likely just a coincidence.

After more debugging today we're pretty sure this is some sort of networking issue between our internal npm server (sinopia) and our build agents, since like you said the process should just exit immediately if we were overflowing the memory limit. And we've had repeated incidents of ECONNRESET with the newly up'd memory limits.

Thanks again for the write-up and adding the feature to your backlog - the work you and the rest the cli team do is greatly appreciated!

@daavve
Copy link

@daavve daavve commented Oct 28, 2016

I use aliasing to pass options to node for npm. Here is my .bashrc:

alias npm='node --max_old_space_size=8000 /usr/bin/npm'

@MichaelJCole
Copy link

@MichaelJCole MichaelJCole commented Nov 20, 2016

@othiym23 I don't think this is an edge use-case. Do I need to hard-code this in my docker image, even though the memory is allocated at runtime? Is there a reason this isn't an environment variable?

@ebdrup
Copy link

@ebdrup ebdrup commented Feb 14, 2017

We nee this quite often, when running large things using npm scripts. npm run XXX. Would love to see a fix for this.

@bchr02
Copy link

@bchr02 bchr02 commented Feb 28, 2017

@othiym23 so is it max_old_space_size or max-old-space-size (dashes or underscores)?

@bchr02
Copy link

@bchr02 bchr02 commented Feb 28, 2017

nevermind, I found it by RTFM here:

--v8-options

Added in: v0.1.3
Print v8 command line options.

Note: v8 options allow words to be separated by both dashes (-) or underscores (_).

For example, --stack-trace-limit is equivalent to --stack_trace_limit.

jorrit-wehelp added a commit to jorrit-wehelp/ionic-app-scripts that referenced this issue Mar 28, 2017
Linux only accept one argument in a shebang, so everything after the
/usr/bin/env is seen as one argument.

For setting this node option, see
npm/npm#12238

Resolve ionic-team#838
danbucholtz added a commit to ionic-team/ionic-app-scripts that referenced this issue Mar 28, 2017
… giving app-scripts more memory by default

Linux only accept one argument in a shebang, so everything after the
/usr/bin/env is seen as one argument.

For setting this node option, see
npm/npm#12238

Resolve #838
@maxgalbu
Copy link

@maxgalbu maxgalbu commented May 2, 2017

There IS an easy way to pass parameter down to node from npm:

npm run something -- --max-old-space-size=250 --anything-you-want

@KenanY
Copy link
Collaborator

@KenanY KenanY commented May 2, 2017

@maxgalbu As far as I know, that just passes those args down to whatever your "something" script is.

@rmorlang
Copy link

@rmorlang rmorlang commented May 16, 2017

I had success with:

node --max-old-space-size=250 `which npm` some_npm_command
@majorcool
Copy link

@majorcool majorcool commented Jun 13, 2017

I faced this issue too, and found this npm package helpful increase-memory-limit

@nodkrot
Copy link

@nodkrot nodkrot commented Jun 28, 2017

Example to previous answer

node --max_old_space_size=2000 `which npm` start
@chamini2
Copy link

@chamini2 chamini2 commented Jun 29, 2017

I would have expected to be able to pass node parameters with npm. Something like (for example):

$ npm run start --node-flags --max-old-space-size=512 --no-warnings
@nikhilo
Copy link

@nikhilo nikhilo commented Jul 10, 2017

@MichaelJCole Were you able to find out a solution for setting the value in Docker container (at run time) ?

@sandorfr
Copy link

@sandorfr sandorfr commented Aug 11, 2017

The package suggested by @EXin it a good workaround for now.

I have the feeling this is mostly a problem with webpack or rollup and they might be running on a child process of the actual process started with the npm run command. For that reason, it might be there responsibility to tweak their commands so people don't run into those issues.

@chamini2
Copy link

@chamini2 chamini2 commented Aug 11, 2017

I have the feeling this is mostly a problem with webpack or rollup and they might be running on a child process of the actual process started with the npm run command. For that reason, it might be there responsibility to tweak their commands so people don't run into those issues.

I disagree, maybe the tweaking is necessary too, but I should be able to pass these flags to node.

@HipsterZipster
Copy link

@HipsterZipster HipsterZipster commented Sep 29, 2017

It'd be even nicer if there was a ENV_VAR to set this for an entire system

@lostpebble
Copy link

@lostpebble lostpebble commented Nov 1, 2017

@HipsterZipster it's not perfect, but you could set it up for yourself. I'm currently doing this, which seems to work well:

scripts: {
    start: "cross-var node --max-old-space-size=$NODE_JS_MAX_OLD_SPACE_SIZE --max-semi-space-size=$NODE_JS_MAX_SEMI_SPACE_SIZE ./server.js",
  },

cross-var is a module which allows you to use environment variables cross-platform in the way I'm using them there.

@kuncevic
Copy link

@kuncevic kuncevic commented Feb 13, 2018

any news if that is coming?
How would you make this node --max_old_space_size=2000 which npm start to work on windows, seems like which = where on windows but wont work replacing just that.

There is one more bash solution https://stackoverflow.com/a/40939496/415078 wonder is there is a way to do something similar on Windows:
node --max_old_space_size=8000 /usr/bin/npm install -g ionic

@mischkl
Copy link
Contributor

@mischkl mischkl commented Feb 20, 2018

desperately needed here too! cross-platform team running into issues with webpack, especially on windows. would be great to be able to specify max-old-space-size via the npm script

@mischkl
Copy link
Contributor

@mischkl mischkl commented Feb 20, 2018

So it turns out that instead of needing to alias npm or otherwise call node directly, you can increase Node's max heap size by setting the NODE_OPTIONS environmental variable (introduced in Node 8) as follows:

NODE_OPTIONS=--max_old_space_size=4096

Usage with NPM scripts:

"scripts": {
    "start": "cross-env NODE_OPTIONS=--max_old_space_size=4096 webpack"
}

Note that it's important to specify the option with_underscores since that's the only one that NODE_OPTIONS accepts.

@noumaans
Copy link

@noumaans noumaans commented Mar 10, 2018

NODE_OPTIONS=--max-old-space-size=4096
Please use hyphens instead of underscores if setting it as an environment variable in your shell or through Makefile.

https://nodejs.org/dist/latest-v8.x/docs/api/cli.html#cli_node_options_options

@mischkl
Copy link
Contributor

@mischkl mischkl commented Mar 11, 2018

It turns out either way should work, see nodejs/node#14093. I didn't see that documentation and was basing my statement on the source code, but I overlooked that either variant is allowed.

@AnyhowStep
Copy link

@AnyhowStep AnyhowStep commented Jun 11, 2018

Is there a way to set the --max-old-space-size to more than 8GB?
I'm trying to type check a TypeScript project but I keep running into out of memory errors.

I tried setting it to 16GB but it still just uses 8GB before giving up.

[EDIT]
At this point, I just want to see how much RAM I can throw at tsc till it compiles. If possible, I'd allocate all 32 GB and then throw in swap space as well.

@huan
Copy link

@huan huan commented Jul 2, 2018

@AnyhowStep I'm interested in how to increase the memory to 32GB too.

Did you get any progress about this?

BTW: it seems that you have a huge TypeScript project!

@AnyhowStep
Copy link

@AnyhowStep AnyhowStep commented Jul 3, 2018

@zixia

It's not so much huge as it is complicated, I think.

I never did figure it out, unfortunately.

@flash-me
Copy link

@flash-me flash-me commented Jul 4, 2018

1. Proposal

You can create a .npmrc in your project (see also project-config-file) and define your options there:
NODE_OPTIONS=--max-old-space-size=4096
npm config ls to see the output.

2. Proposal

You add a config field in your package.json and define your options there.

I'll try out both of this myself. I hope they work, since these would a much cleaner approach instead of writing "node ..." in front of every npm script...
And to ensure that everyone has the same project settings without touching userenvironment config.

Edit

Both does not work. I couldn't find any solution to set this value within project configurations. There is neither the possibility to run a wildcard script before any npm script. Either you have to set the user environment variable (which you can't access for good reasons within npm) or you have to insert it in all npm scripts, which is quite ugly...

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.