Repository navigation
Automatically restart processes using more than a fixed amount of RAM #141
Description
Activity
I need it too. Maybe restart based on amount of CPU usage too.
Reacted by Paul Deng and Samuel ThompsonReacted by Prasanth+1
👍 Very useful feature for production while debugging the leak in dev.
+1
+1 , since PM2 cluster_mode takes much memory. if it can be restart when memory reached 500MB like will be great. it takes around 1.5GB when we keep process 3,4 days :(
I think you could set a memory limit in node.js/v8 command-line arguments, so it'll crash when the limit is reached. And pm2 will restart it as usual. Would this work for you?
I'm talking about
node --max-old-space-size=XXX test.js, with which a process that eats too much will die with "FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory".If you're using pm2, I suppose
pm2 start --node-args="--max-old-space-size=XXX" test.jswill give the same effect.I'm not sure if it's the right way, but it seems worth exploring.
Reacted by Jeffers D, TNT-Likely, Nimatullah Razmjo, yairyairyair and Taylor EdmistonReacted by Thomas and Nimatullah RazmjoReacted by Alice Hernandez and Nimatullah Razmjo4 remaining items
I've added the option --max-restart-memory: https://github.com/Unitech/pm2/tree/development#automatic-restart-process-based-on-memory
Thanks for that v8 trick @rlidwka !
Reacted by Rocco Musolino, Ilmari Oranen and AnkeshI have the same problem where after about 8-10 hours, the pm2 instance runs of out of memory and never recovers. When I log into the server and run pm2 list, its like pm2 was never running and it restarts the Daemonizer. Here's my log. This is running on a 512MB Debian digital ocean droplet. I run it with the command 'pm2 start -x index.js'
2015-04-10 21:02:42: Starting execution sequence in -fork mode- for app name:index id:0 2015-04-10 21:02:42: App name:index id:0 online 2015-04-11 06:50:00: ERROR CAUGHT BY DOMAIN: Error: spawn ENOMEM at exports._errnoException (util.js:746:11) at ChildProcess.spawn (child_process.js:1155:11) at exports.spawn (child_process.js:988:9) at Object.exports.execFile (child_process.js:682:15) at exports.exec (child_process.js:642:18) at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5) at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34) at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34) at processImmediate [as _immediateCallback] (timers.js:358:17) child_process.js:1155 throw errnoException(err, 'spawn'); ^ Error: spawn ENOMEM at exports._errnoException (util.js:746:11) at ChildProcess.spawn (child_process.js:1155:11) at exports.spawn (child_process.js:988:9) at Object.exports.execFile (child_process.js:682:15) at exports.exec (child_process.js:642:18) at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5) at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34) at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34) at processImmediate [as _immediateCallback] (timers.js:358:17) 2015-04-13 15:54:37: [PM2][WORKER] Started with refreshing interval: 30000 2015-04-13 15:54:37: [[[[ PM2/God daemon launched ]]]] 2015-04-13 15:54:37: BUS system [READY] on port /root/.pm2/pub.sock 2015-04-13 15:54:37: RPC interface [READY] on port /root/.pm2/rpc.sock 2015-04-13 15:58:24: Starting execution sequence in -fork mode- for app name:index id:0 2015-04-13 15:58:24: App name:index id:0 onlineMaybe it is your app which takes large amounts of memory and thus PM2 doesn't have enough.
Sure but shouldn't pm2 be catching that error and restart itself? Is it because I'm using fork mode? I guess I don't have to since it is only 1 core.. Its using jsdom and jquery and reading web page, I make sure to close the session each time but I'm sure its possible the dom library has some leaks in it.
Idk if its a PM2 issue or a Node issue in general, but every app I've made I've noticed that memory use always constantly increases, its never dumped. Is there a fix for this that I've missed?
Well, the thing is, in order to update itself, PM2 calls spawn('pm2 update') but since there's not enough memory for spawn(), it doesn't work. That's why you see the error twice. The first time it gets triggered by a spawn() in the worker, the second time it gets triggered when pm2 tries to update itself.
I think 512MB of RAM is just not enough nowadays to run your apps + a process manager while both of them use Node.js (which means being run by a VM such as Google V8 Runtime therefore using a lot of memory) :/It's just a worker which checks every 30sec the memory usage of your app and restarts it if it exceeds the limit
@ablankenship10 PM2 v0.12.16 is released. This bug shouldn't happen anymore. At least it won't crash PM2 if ENOMEM occurs.
What is the default value of max-memory-restart?
We are seeing that in a nodejs app we have given the max-old-space to be 512M its not hitting that limit. Still pm2 is retsrating our app.
Does pm2 restart on cpu reaching 100% or anything too?
Does pm2 restart on cpu reaching 100% or anything too?
Nope.What is the default value of max-memory-restart?
There is none, it is the nodejs default max-old-space-sizeBtw, in nodejs 7 I see:
--max_old_space_size (max size of the old space (in Mbytes)) type: int default: 0
It would be very valuable to get the option to kill and respawn processes that are using more memory than a configurable amount. This would help a lot in the case of a leaking app. Indeed, consider the following scenario:
I'm not saying that leaking apps are a good thing. But from a "host" point of view, killing and respawning seems like the best thing to do. This is what Heroku does, for example.