PM2 seems to die on ressurect

[null4bl3]

Running a 4 process clustered instance with PM2. This morning we had a failure where the PM2 God deamon had died. Logging through the agent.log file i see these entries:

[WATCHDOG] PM2 is disconnected - Relaunching PM2

Followed by:

Error: Cannot find module '~/.pm2/logs/bin/pm2'

And ending with the following errors for a ECONNREFUSED for a keymetrics subdomain.

[REV] connect ECONNREFUSED 195.154.156.78:43554                                                                                   
[REV] connect ECONNREFUSED 195.154.156.78:43554                                                                                   
[REV] Connected to cassiope.keymetrics.io:43554                                                                                   Buffer size too high (2792762), stopping buffering and sending                                                                    
Buffer size too high (3286842), stopping buffering and sending                                                                    
[WARN] Time to send data over TCP took 1.084seconds!                                                                              
Buffer size too high (2933066), stopping buffering and sending

Looking at the Watchdog.js file i find that PM2 on ressurect is trying to run the following command:

'node ' + process.cwd() + '/bin/pm2 resurrect'

A command i honestly can't see should ever work as PM2 is not installed, atleast on a unix system under /bin/pm2 or in a projects root.

[Unitech]

[WATCHDOG] PM2 is disconnected - Relaunching PM2 means that the agent lost connection to PM2 (in case of PM2 crash) and the agent re spawned PM2 with the previous process list.

About the 'node ' + process.cwd() + '/bin/pm2 resurrect', how did you installed PM2? Where is it located?
Also could you show me the pm2.log file?

[null4bl3]

PM2 was installed using

npm install -g pm2

Placing pm2 at: /usr/local/bin/pm2 on our Ubuntu server
The pm2.log file contains the following:

===============================================================================
--- New PM2 Daemon started ----------------------------------------------------
Time                 : Tue Sep 12 2017 06:19:27 GMT+0000 (UTC)
PM2 version          : 2.5.0
Node.js version      : 8.1.3
Current arch         : x64
PM2 home             : /home/xxxxxxxxxxxxxx/.pm2
PM2 PID file         : /home/xxxxxxxxxxxxxx/.pm2/pm2.pid
RPC socket file      : /home/xxxxxxxxxxxxxx/.pm2/rpc.sock
BUS socket file      : /home/xxxxxxxxxxxxxx/.pm2/pub.sock
Application log path : /home/xxxxxxxxxxxxxx/.pm2/logs
Process dump file    : /home/xxxxxxxxxxxxxx/.pm2/dump.pm2
Concurrent actions   : 2
SIGTERM timeout      : 1600
===============================================================================
Starting execution sequence in -fork mode- for app name:pm2-server-monit id:0
App name:pm2-server-monit id:0 online
Starting execution sequence in -fork mode- for app name:pm2-mysql id:1
App name:pm2-mysql id:1 online
Starting execution sequence in -cluster mode- for app name:XXXv4 id:2
Starting execution sequence in -cluster mode- for app name:XXXv4 id:3
App name:XXXv4 id:2 online
(node:30594) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.
Starting execution sequence in -cluster mode- for app name:XXXv4 id:4
App name:XXXv4 id:3 online
(node:30594) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.
Starting execution sequence in -cluster mode- for app name:XXXv4 id:5
App name:XXXv4 id:4 online
(node:30594) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.
App name:XXXv4 id:5 online
(node:30594) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.
Starting execution sequence in -fork mode- for app name:marienlund id:6
Starting execution sequence in -cluster mode- for app name:stage id:7
App name:marienlund id:6 online
Starting execution sequence in -cluster mode- for app name:stage id:8
App name:stage id:7 online
(node:30594) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.
Starting execution sequence in -fork mode- for app name:xxxx1 id:9
App name:xxxx1 id:9 online
App name:stage id:8 online
(node:30594) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.

[Unitech]

Placing pm2 at: /usr/local/bin/pm2 on our Ubuntu serverdo you mean you copied pm2? Or built a symbolic link?

[null4bl3]

Sorry. PM2 was installed globally using npm.
This installed PM2 in the directory:

/usr/local/lib/node_modules

we did not copy the files there or build a symbolic link. Just installed via NPM using the global parameter.

[null4bl3]

We just had it happen again today in production.
A massive flood of entries in agent.log with the likes of:

2017-09-14 11:04:57: Buffer size too high (5551194), stopping buffering and sending
2017-09-14 11:04:57: [PUSH] Forcing reconnection
2017-09-14 11:04:57: [PUSH] Connecting cassiope.keymetrics.io:80
2017-09-14 11:04:57: [PUSH] Connected
2017-09-14 11:05:50: Buffer size too high (7635082), stopping buffering and sending
2017-09-14 11:05:50: [WARN] Time to send data over TCP took 1.611seconds!
2017-09-14 11:06:50: Buffer size too high (8026266), stopping buffering and sending

Followed by a long list of:

Error: Cannot find module '/xxx/xxx/xxx/prod/bin/pm2'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

2017-09-14 12:47:47: [WATCHDOG] PM2 is disconnected - Relaunching PM2
2017-09-14 12:47:47: [WATCHDOG] Trying to launch PM2 #1
2017-09-14 12:47:47: [WATCHDOG] PM2 is disconnected - Relaunching PM2
2017-09-14 12:47:47: [WATCHDOG] Already relaunching PM2
module.js:487 throw err;

Again it is attempting to find the PM2 executable in the root folder of our project, which to me indicates that the Watchdog.js file path to resurrect found here point to the project root through process.cwd().

Any thoughts on that theory?

[Unitech]

Which node.js version are you using?

[Unitech]

Ok I managed to reproduce the issue
Working on it.

[Unitech]

Fixed on recently published pm2 (2.7.0):

$ npm install pm2@latest -g
$ pm2 update

Let me know if there is any issue,