New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically restart processes using more than a fixed amount of RAM #141

Closed
Ecco opened this Issue Nov 1, 2013 · 24 comments

Comments

@Ecco

Ecco commented Nov 1, 2013

It would be very valuable to get the option to kill and respawn processes that are using more memory than a configurable amount. This would help a lot in the case of a leaking app. Indeed, consider the following scenario:

  • No respawn: The leaking app ends up taking tons of memory, effectively bringing the server down
  • Auto respawn: It might indeed require a bit of CPU time, but the service can be kept up and running

I'm not saying that leaking apps are a good thing. But from a "host" point of view, killing and respawning seems like the best thing to do. This is what Heroku does, for example.

@rodrigok

This comment has been minimized.

Show comment
Hide comment
@rodrigok

rodrigok Nov 4, 2013

I need it too. Maybe restart based on amount of CPU usage too.

rodrigok commented Nov 4, 2013

I need it too. Maybe restart based on amount of CPU usage too.

@abelaska

This comment has been minimized.

Show comment
Hide comment
@abelaska

abelaska commented Nov 18, 2013

+1

@dandv

This comment has been minimized.

Show comment
Hide comment
@dandv

dandv Mar 7, 2014

Contributor

👍 Very useful feature for production while debugging the leak in dev.

Contributor

dandv commented Mar 7, 2014

👍 Very useful feature for production while debugging the leak in dev.

@tjwebb

This comment has been minimized.

Show comment
Hide comment
@tjwebb

tjwebb commented Mar 27, 2014

+1

@devsmart

This comment has been minimized.

Show comment
Hide comment
@devsmart

devsmart Jul 5, 2014

+1 , since PM2 cluster_mode takes much memory. if it can be restart when memory reached 500MB like will be great. it takes around 1.5GB when we keep process 3,4 days :(

image

devsmart commented Jul 5, 2014

+1 , since PM2 cluster_mode takes much memory. if it can be restart when memory reached 500MB like will be great. it takes around 1.5GB when we keep process 3,4 days :(

image

@rlidwka

This comment has been minimized.

Show comment
Hide comment
@rlidwka

rlidwka Jul 16, 2014

Collaborator

I think you could set a memory limit in node.js/v8 command-line arguments, so it'll crash when the limit is reached. And pm2 will restart it as usual. Would this work for you?

Collaborator

rlidwka commented Jul 16, 2014

I think you could set a memory limit in node.js/v8 command-line arguments, so it'll crash when the limit is reached. And pm2 will restart it as usual. Would this work for you?

@devsmart

This comment has been minimized.

Show comment
Hide comment
@devsmart

devsmart Jul 17, 2014

Thanks @rlidwka you mean by V8_MAX_SEMISPACE_SIZE ? or any other way to set it on with PM2 ?
I usually use simple PM2 command "pm2 start app.js -i max" appreciate if you can explain more how I can set memory limit to my application.

devsmart commented Jul 17, 2014

Thanks @rlidwka you mean by V8_MAX_SEMISPACE_SIZE ? or any other way to set it on with PM2 ?
I usually use simple PM2 command "pm2 start app.js -i max" appreciate if you can explain more how I can set memory limit to my application.

@soyuka soyuka added mid-priority and removed low-priority labels Jul 17, 2014

@rlidwka

This comment has been minimized.

Show comment
Hide comment
@rlidwka

rlidwka Jul 17, 2014

Collaborator

I'm talking about node --max-old-space-size=XXX test.js, with which a process that eats too much will die with "FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory".

If you're using pm2, I suppose pm2 start --node-args="--max-old-space-size=XXX" test.js will give the same effect.

I'm not sure if it's the right way, but it seems worth exploring.

Collaborator

rlidwka commented Jul 17, 2014

I'm talking about node --max-old-space-size=XXX test.js, with which a process that eats too much will die with "FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory".

If you're using pm2, I suppose pm2 start --node-args="--max-old-space-size=XXX" test.js will give the same effect.

I'm not sure if it's the right way, but it seems worth exploring.

@soyuka

This comment has been minimized.

Show comment
Hide comment
@soyuka

soyuka Jul 17, 2014

Collaborator

That's nice and working.

Just tested with a small memory leak that tests the pidusage module. This test breaks when memory is up to 200 MB and with --max-old-space-size=100 it breaks at 100 MB:

FATAL ERROR: JS Allocation failed - process out of memory
[1]    5718 abort      node --max-old-space-size=100 test/stresstest.js

Of course when using pm2 the process will be restarted.

Thanks @rlidwka for this nice information!

Should it be added to pm2 anyway?

To restart on CPU or memory amount with pm2 we 'd have to check the process status based on an interval which would lead to more CPU usage, I don't think that it's a good idea. Suggestions are welcome.

Collaborator

soyuka commented Jul 17, 2014

That's nice and working.

Just tested with a small memory leak that tests the pidusage module. This test breaks when memory is up to 200 MB and with --max-old-space-size=100 it breaks at 100 MB:

FATAL ERROR: JS Allocation failed - process out of memory
[1]    5718 abort      node --max-old-space-size=100 test/stresstest.js

Of course when using pm2 the process will be restarted.

Thanks @rlidwka for this nice information!

Should it be added to pm2 anyway?

To restart on CPU or memory amount with pm2 we 'd have to check the process status based on an interval which would lead to more CPU usage, I don't think that it's a good idea. Suggestions are welcome.

@devsmart

This comment has been minimized.

Show comment
Hide comment
@devsmart

devsmart Jul 18, 2014

Thanks a lot @rlidwka !

devsmart commented Jul 18, 2014

Thanks a lot @rlidwka !

@soyuka soyuka closed this Aug 13, 2014

@soyuka soyuka added wontfix and removed enhancement labels Aug 13, 2014

@Unitech

This comment has been minimized.

Show comment
Hide comment
@Unitech

Unitech Aug 14, 2014

Owner

I've added the option --max-restart-memory: https://github.com/Unitech/pm2/tree/development#automatic-restart-process-based-on-memory

Thanks for that v8 trick @rlidwka !

Owner

Unitech commented Aug 14, 2014

I've added the option --max-restart-memory: https://github.com/Unitech/pm2/tree/development#automatic-restart-process-based-on-memory

Thanks for that v8 trick @rlidwka !

@yorkie

This comment has been minimized.

Show comment
Hide comment
@yorkie

yorkie Apr 13, 2015

Hey @Unitech actually the max-restart-memory doesn't work well in my machine, the memory in my machine keeps under 100M, but my node instance will automatically restart every 30s.

yorkie commented Apr 13, 2015

Hey @Unitech actually the max-restart-memory doesn't work well in my machine, the memory in my machine keeps under 100M, but my node instance will automatically restart every 30s.

@soyuka

This comment has been minimized.

Show comment
Hide comment
@soyuka

soyuka Apr 13, 2015

Collaborator

@yorkie please see #1159.
Could you show your pm2.log? This might be an issue in your app.

Collaborator

soyuka commented Apr 13, 2015

@yorkie please see #1159.
Could you show your pm2.log? This might be an issue in your app.

@ablankenship10

This comment has been minimized.

Show comment
Hide comment
@ablankenship10

ablankenship10 Apr 13, 2015

I have the same problem where after about 8-10 hours, the pm2 instance runs of out of memory and never recovers. When I log into the server and run pm2 list, its like pm2 was never running and it restarts the Daemonizer. Here's my log. This is running on a 512MB Debian digital ocean droplet. I run it with the command 'pm2 start -x index.js'

2015-04-10 21:02:42: Starting execution sequence in -fork mode- for app name:index id:0
2015-04-10 21:02:42: App name:index id:0 online
2015-04-11 06:50:00: ERROR CAUGHT BY DOMAIN:
Error: spawn ENOMEM
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1155:11)
    at exports.spawn (child_process.js:988:9)
    at Object.exports.execFile (child_process.js:682:15)
    at exports.exec (child_process.js:642:18)
    at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5)
    at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
    at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
    at processImmediate [as _immediateCallback] (timers.js:358:17)
child_process.js:1155
    throw errnoException(err, 'spawn');
          ^
Error: spawn ENOMEM
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1155:11)
    at exports.spawn (child_process.js:988:9)
    at Object.exports.execFile (child_process.js:682:15)
    at exports.exec (child_process.js:642:18)
    at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5)
    at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
    at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
    at processImmediate [as _immediateCallback] (timers.js:358:17)
2015-04-13 15:54:37: [PM2][WORKER] Started with refreshing interval: 30000
2015-04-13 15:54:37: [[[[ PM2/God daemon launched ]]]]
2015-04-13 15:54:37: BUS system [READY] on port /root/.pm2/pub.sock
2015-04-13 15:54:37: RPC interface [READY] on port /root/.pm2/rpc.sock
2015-04-13 15:58:24: Starting execution sequence in -fork mode- for app name:index id:0
2015-04-13 15:58:24: App name:index id:0 online

ablankenship10 commented Apr 13, 2015

I have the same problem where after about 8-10 hours, the pm2 instance runs of out of memory and never recovers. When I log into the server and run pm2 list, its like pm2 was never running and it restarts the Daemonizer. Here's my log. This is running on a 512MB Debian digital ocean droplet. I run it with the command 'pm2 start -x index.js'

2015-04-10 21:02:42: Starting execution sequence in -fork mode- for app name:index id:0
2015-04-10 21:02:42: App name:index id:0 online
2015-04-11 06:50:00: ERROR CAUGHT BY DOMAIN:
Error: spawn ENOMEM
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1155:11)
    at exports.spawn (child_process.js:988:9)
    at Object.exports.execFile (child_process.js:682:15)
    at exports.exec (child_process.js:642:18)
    at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5)
    at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
    at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
    at processImmediate [as _immediateCallback] (timers.js:358:17)
child_process.js:1155
    throw errnoException(err, 'spawn');
          ^
Error: spawn ENOMEM
    at exports._errnoException (util.js:746:11)
    at ChildProcess.spawn (child_process.js:1155:11)
    at exports.spawn (child_process.js:988:9)
    at Object.exports.execFile (child_process.js:682:15)
    at exports.exec (child_process.js:642:18)
    at getUrl (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/vizion/lib/git.js:15:5)
    at fn (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:641:34)
    at Immediate._onImmediate (/root/.nvm/versions/node/v0.12.0/lib/node_modules/pm2/node_modules/async/lib/async.js:557:34)
    at processImmediate [as _immediateCallback] (timers.js:358:17)
2015-04-13 15:54:37: [PM2][WORKER] Started with refreshing interval: 30000
2015-04-13 15:54:37: [[[[ PM2/God daemon launched ]]]]
2015-04-13 15:54:37: BUS system [READY] on port /root/.pm2/pub.sock
2015-04-13 15:54:37: RPC interface [READY] on port /root/.pm2/rpc.sock
2015-04-13 15:58:24: Starting execution sequence in -fork mode- for app name:index id:0
2015-04-13 15:58:24: App name:index id:0 online
@jshkurti

This comment has been minimized.

Show comment
Hide comment
@jshkurti

jshkurti Apr 13, 2015

Contributor

Maybe it is your app which takes large amounts of memory and thus PM2 doesn't have enough.

Contributor

jshkurti commented Apr 13, 2015

Maybe it is your app which takes large amounts of memory and thus PM2 doesn't have enough.

@ablankenship10

This comment has been minimized.

Show comment
Hide comment
@ablankenship10

ablankenship10 Apr 13, 2015

Sure but shouldn't pm2 be catching that error and restart itself? Is it because I'm using fork mode? I guess I don't have to since it is only 1 core.. Its using jsdom and jquery and reading web page, I make sure to close the session each time but I'm sure its possible the dom library has some leaks in it.

Idk if its a PM2 issue or a Node issue in general, but every app I've made I've noticed that memory use always constantly increases, its never dumped. Is there a fix for this that I've missed?

ablankenship10 commented Apr 13, 2015

Sure but shouldn't pm2 be catching that error and restart itself? Is it because I'm using fork mode? I guess I don't have to since it is only 1 core.. Its using jsdom and jquery and reading web page, I make sure to close the session each time but I'm sure its possible the dom library has some leaks in it.

Idk if its a PM2 issue or a Node issue in general, but every app I've made I've noticed that memory use always constantly increases, its never dumped. Is there a fix for this that I've missed?

@jshkurti

This comment has been minimized.

Show comment
Hide comment
@jshkurti

jshkurti Apr 13, 2015

Contributor

Well, the thing is, in order to update itself, PM2 calls spawn('pm2 update') but since there's not enough memory for spawn(), it doesn't work. That's why you see the error twice. The first time it gets triggered by a spawn() in the worker, the second time it gets triggered when pm2 tries to update itself.
I think 512MB of RAM is just not enough nowadays to run your apps + a process manager while both of them use Node.js (which means being run by a VM such as Google V8 Runtime therefore using a lot of memory) :/

Contributor

jshkurti commented Apr 13, 2015

Well, the thing is, in order to update itself, PM2 calls spawn('pm2 update') but since there's not enough memory for spawn(), it doesn't work. That's why you see the error twice. The first time it gets triggered by a spawn() in the worker, the second time it gets triggered when pm2 tries to update itself.
I think 512MB of RAM is just not enough nowadays to run your apps + a process manager while both of them use Node.js (which means being run by a VM such as Google V8 Runtime therefore using a lot of memory) :/

@yorkie

This comment has been minimized.

Show comment
Hide comment
@yorkie

yorkie Apr 13, 2015

Thanks guys, my issue is addressed by restarted the pm2 process, because in my previous boot script I set max-memory-restart to 80m, but later i changed it to 150m, but I guess there is a cache problem, so the solution is to kill the pm2 and restart it by using the new setup.

By the way, could someone can point out how the max-memory-restart is working? what's the difference from the v8 flag: max-old-space-size?

yorkie commented Apr 13, 2015

Thanks guys, my issue is addressed by restarted the pm2 process, because in my previous boot script I set max-memory-restart to 80m, but later i changed it to 150m, but I guess there is a cache problem, so the solution is to kill the pm2 and restart it by using the new setup.

By the way, could someone can point out how the max-memory-restart is working? what's the difference from the v8 flag: max-old-space-size?

@jorge-d

This comment has been minimized.

Show comment
Hide comment
@jorge-d

jorge-d Apr 14, 2015

Contributor

@yorkie All the magic is happening here :)

Contributor

jorge-d commented Apr 14, 2015

@yorkie All the magic is happening here :)

@yorkie

This comment has been minimized.

Show comment
Hide comment
@yorkie

yorkie Apr 14, 2015

Okay thank you @jorge-d

yorkie commented Apr 14, 2015

Okay thank you @jorge-d

@jshkurti

This comment has been minimized.

Show comment
Hide comment
@jshkurti

jshkurti Apr 14, 2015

Contributor

It's just a worker which checks every 30sec the memory usage of your app and restarts it if it exceeds the limit

Contributor

jshkurti commented Apr 14, 2015

It's just a worker which checks every 30sec the memory usage of your app and restarts it if it exceeds the limit

@jshkurti

This comment has been minimized.

Show comment
Hide comment
@jshkurti

jshkurti Jun 5, 2015

Contributor

@ablankenship10 PM2 v0.12.16 is released. This bug shouldn't happen anymore. At least it won't crash PM2 if ENOMEM occurs.

https://github.com/Unitech/PM2/releases/tag/0.12.16

Contributor

jshkurti commented Jun 5, 2015

@ablankenship10 PM2 v0.12.16 is released. This bug shouldn't happen anymore. At least it won't crash PM2 if ENOMEM occurs.

https://github.com/Unitech/PM2/releases/tag/0.12.16

@Unitech Unitech removed the wontfix label Aug 27, 2015

@akamanocha

This comment has been minimized.

Show comment
Hide comment
@akamanocha

akamanocha Nov 24, 2016

What is the default value of max-memory-restart?

We are seeing that in a nodejs app we have given the max-old-space to be 512M its not hitting that limit. Still pm2 is retsrating our app.

Does pm2 restart on cpu reaching 100% or anything too?

akamanocha commented Nov 24, 2016

What is the default value of max-memory-restart?

We are seeing that in a nodejs app we have given the max-old-space to be 512M its not hitting that limit. Still pm2 is retsrating our app.

Does pm2 restart on cpu reaching 100% or anything too?

@soyuka

This comment has been minimized.

Show comment
Hide comment
@soyuka

soyuka Nov 24, 2016

Collaborator

Does pm2 restart on cpu reaching 100% or anything too?
Nope.

What is the default value of max-memory-restart?
There is none, it is the nodejs default max-old-space-size

Btw, in nodejs 7 I see:

  --max_old_space_size (max size of the old space (in Mbytes))
        type: int  default: 0
Collaborator

soyuka commented Nov 24, 2016

Does pm2 restart on cpu reaching 100% or anything too?
Nope.

What is the default value of max-memory-restart?
There is none, it is the nodejs default max-old-space-size

Btw, in nodejs 7 I see:

  --max_old_space_size (max size of the old space (in Mbytes))
        type: int  default: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment