pm2: Daemon running at 100% CPU

[randomsock]

On CentOS 6, pm2 versions 0.10.5 and 0.10.7 at least (I don't know exactly when it started), pm2: Daemon is constantly spinning at 100% CPU. This is happening on at least 2 different servers so I can't see it's something specific, though we are running pm2-web and node-inspector in both, but apart from that just different web apps. Tried all the usual restarts and reboots.

[Unitech]

Can you show PM2 logs (tail -f ~/.pm2/pm2.log) ?
Process has been starrted in fork mode or cluster mode ?
Node.js version ?

[randomsock]

All processes clustered, but it's the Daemon itself that's spinning.
node 0.10.10
pm2.log:
[[[[ PM2/God daemon launched ]]]]
RPC interface [READY] on port 6666
BUS system [READY] on port 6667
Entering in node wrap logic (cluster_mode) for script /usr/local/bin/pm2-web
/usr/local/bin/pm2-web - id0 worker online
Entering in node wrap logic (cluster_mode) for script /usr/local/bin/node-inspector
/usr/local/bin/node-inspector - id1 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id2 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id3 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id4 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id5 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/blah/src/server.js
/blah/blah/blah/blah/src/server.js - id6 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/server.js
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/server.js
process with pid 2094 successfully killed
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/server.js
process with pid 2468 successfully killed
/blah/blah/blah/server.js - id7 worker online

[trawor]

+1

Ubuntu, pm2 v0.8.x

several time, eat up 100% CPU

-- 
Travis

在 2014年9月15日 下午5:55:22, randomsock (notifications@github.com) 写到:

All processes clustered, but it's the Daemon itself that's spinning.
node 0.10.10
pm2.log:
[[[[ PM2/God daemon launched ]]]]
RPC interface [READY] on port 6666
BUS system [READY] on port 6667
Entering in node wrap logic (cluster_mode) for script /usr/local/bin/pm2-web
/usr/local/bin/pm2-web - id0 worker online
Entering in node wrap logic (cluster_mode) for script /usr/local/bin/node-inspector
/usr/local/bin/node-inspector - id1 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id2 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id3 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id4 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id5 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/blah/src/server.js
/blah/blah/blah/blah/src/server.js - id6 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/server.js
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/server.js
process with pid 2094 successfully killed
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script /blah/blah/blah/server.js
process with pid 2468 successfully killed
/blah/blah/blah/server.js - id7 worker online

—
Reply to this email directly or view it on GitHub.

[Unitech]

After a pm2 updatePM2 is it still the same?
On Sep 15, 2014 11:57 AM, "Travis" notifications@github.com wrote:

+1

Ubuntu, pm2 v0.8.x

several time, eat up 100% CPU

Travis

在 2014年9月15日 下午5:55:22, randomsock (notifications@github.com) 写到:

All processes clustered, but it's the Daemon itself that's spinning.
node 0.10.10
pm2.log:
[[[[ PM2/God daemon launched ]]]]
RPC interface [READY] on port 6666
BUS system [READY] on port 6667
Entering in node wrap logic (cluster_mode) for script
/usr/local/bin/pm2-web
/usr/local/bin/pm2-web - id0 worker online
Entering in node wrap logic (cluster_mode) for script
/usr/local/bin/node-inspector
/usr/local/bin/node-inspector - id1 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id2 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id3 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id4 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id5 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/blah/src/server.js
/blah/blah/blah/blah/src/server.js - id6 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/server.js
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/server.js
process with pid 2094 successfully killed
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/server.js
process with pid 2468 successfully killed
/blah/blah/blah/server.js - id7 worker online

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#710 (comment).

[trawor]

I updated to 0.10.7(from 0.8.x), and will keep an eye on it

-- 
Travis
http://fir.im

在 2014年9月15日 下午6:05:46, Alexandre Strzelewicz (notifications@github.com) 写到:

After a pm2 updatePM2 is it still the same?
On Sep 15, 2014 11:57 AM, "Travis" notifications@github.com wrote:

+1

Ubuntu, pm2 v0.8.x

several time, eat up 100% CPU

Travis

在 2014年9月15日 下午5:55:22, randomsock (notifications@github.com) 写到:

All processes clustered, but it's the Daemon itself that's spinning.
node 0.10.10
pm2.log:
[[[[ PM2/God daemon launched ]]]]
RPC interface [READY] on port 6666
BUS system [READY] on port 6667
Entering in node wrap logic (cluster_mode) for script
/usr/local/bin/pm2-web
/usr/local/bin/pm2-web - id0 worker online
Entering in node wrap logic (cluster_mode) for script
/usr/local/bin/node-inspector
/usr/local/bin/node-inspector - id1 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id2 worker online
Entering in node wrap logic (cluster_mode) for script /blah/blah/server.js
/blah/blah/server.js - id3 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id4 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/node_modules/blah/server.js
/blah/blah/blah/node_modules/blah/server.js - id5 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/blah/src/server.js
/blah/blah/blah/blah/src/server.js - id6 worker online
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/server.js
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/server.js
process with pid 2094 successfully killed
/blah/blah/blah/server.js - id7 worker online
Script /blah/blah/blah/server.js 7 exit
Entering in node wrap logic (cluster_mode) for script
/blah/blah/blah/server.js
process with pid 2468 successfully killed
/blah/blah/blah/server.js - id7 worker online

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#710 (comment).

—
Reply to this email directly or view it on GitHub.

[randomsock]

Yes, same after updatePM2 still

[Unitech]

All processes are clustered, in fork mode is it still the same ?

[randomsock]

Yes still the same in fork mode.

Also I pm2 kill'd it, checked everything was gone, then just pm2 ping'd to start the Daemon - it was idle when on its own. As soon as I start any process (cluster or fork) the Daemon spins at 100%. Interestingly, when you stop all processes, the Daemon goes idle again.

[Unitech]

Node.js version ? Can you contact me by email for further inspection https://github.com/Unitech

[trawor]

Exactly what I found out on my server

you can kill pm2 by pid to get rip of this, while pm2 kill can not kill the zombie pm2 process,  check it with ps aux | grep pm2 

-- 
Travis
http://fir.im

在 2014年9月15日 下午6:40:38, randomsock (notifications@github.com) 写到:

Yes still the same in fork mode.

Also I pm2 kill'd it, checked everything was gone, then just pm2 ping'd to start the Daemon - it was idle when on its own. As soon as I start any process (cluster or fork) the Daemon spins at 100%. Interestingly, when you stop all processes, the Daemon goes idle again.

—
Reply to this email directly or view it on GitHub.

[Unitech]

Can you kill pm2 (with pm2 kill or kill command) then start it again like
that:

$ DEBUG='*' pm2 start [...]

And see on the pm2.log what is happening?
On Sep 15, 2014 1:27 PM, "Travis" notifications@github.com wrote:

Exactly what I found out on my server

you can kill pm2 by pid to get rip of this, while pm2 kill can not kill
the zombie pm2 process, check it with ps aux | grep pm2

Travis
http://fir.im

在 2014年9月15日 下午6:40:38, randomsock (notifications@github.com) 写到:

Yes still the same in fork mode.

Also I pm2 kill'd it, checked everything was gone, then just pm2 ping'd to
start the Daemon - it was idle when on its own. As soon as I start any
process (cluster or fork) the Daemon spins at 100%. Interestingly, when you
stop all processes, the Daemon goes idle again.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#710 (comment).

[sketchpixy]

Yup! We are having this issue. 100% CPU. Here are the logs from pm2.log running with DEBUG off:
https://gist.github.com/sketchpixy/5af9893a4e14bc08573e

Output with DEBUG enabled:
https://gist.github.com/sketchpixy/f693674316340e4f2643

pm2.log with DEBUG enabled:
https://gist.github.com/sketchpixy/ab2893f6dd3ff8b23ecf

Node version: v0.10.25
pm2 version: 0.10.5

Will try with updatePM2 and exec_mode: fork_mode and get back to you.

[sketchpixy]

Did updatePM2 (bumped pm2 version to 0.10.7 before running) and changed to fork_mode. Still 100% CPU.

[Unitech]

Are you also on CentOS ?

[sketchpixy]

No its Ubuntu 14.04 LTS.

Currently i'm limiting the CPU usage to 5% (I'm assuming the daemon only monitors the process right?)

$ cpulimit -p $(cat ~/.pm2/pm2.pid) -l 5 -b