PM2 global error caught #4627

Snapper77 · 2020-03-09T12:52:05Z

What's going wrong?

Stop all pm2 apps. (Restarted)

How could we reproduce this issue?

Checking from logs?

Supporting information

PM2 version          : 4.2.0
Node.js version      : 12.13.1
Debian 10 x64

Log details

2020-03-09T18:43:42: PM2 log: ===============================================================================
2020-03-09T18:43:42: PM2 log: --- PM2 global error caught ---------------------------------------------------
2020-03-09T18:43:42: PM2 log: Time                 : Mon Mar 09 2020 18:43:42 GMT+0700 (Western Indonesia Time)
2020-03-09T18:43:42: PM2 error: Cannot read property 'writable' of undefined
2020-03-09T18:43:42: PM2 error: TypeError: Cannot read property 'writable' of undefined
    at PubSocket.send (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:47:14)
    at EventEmitter.<anonymous> (/usr/lib/node_modules/pm2/lib/Daemon.js:445:14)
    at EventEmitter.emit (/usr/lib/node_modules/pm2/node_modules/eventemitter2/lib/eventemitter2.js:330:22)
    at Socket.forkOutData (/usr/lib/node_modules/pm2/lib/God/ForkMode.js:193:17)
    at Socket.emit (events.js:210:5)
    at Socket.EventEmitter.emit (domain.js:498:23)
    at addChunk (_stream_readable.js:309:12)
    at readableAddChunk (_stream_readable.js:290:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at Pipe.onStreamRead (internal/stream_base_commons.js:182:23)
2020-03-09T18:43:42: PM2 log: ===============================================================================
2020-03-09T18:43:42: PM2 error: [PM2] Resurrecting PM2
internal/modules/cjs/loader.js:800
    throw err;
    ^

Error: Cannot find module '/undefined'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:797:15)
    at Function.Module._load (internal/modules/cjs/loader.js:690:27)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1047:10)
    at internal/main/run_main_module.js:17:11 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
2020-03-09T18:43:42: PM2 log: PM2 successfully forked
2020-03-09T18:43:44: PM2 log: ===============================================================================
2020-03-09T18:43:44: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2020-03-09T18:43:44: PM2 log: Time                 : Mon Mar 09 2020 18:43:44 GMT+0700 (Western Indonesia Time)
2020-03-09T18:43:44: PM2 log: PM2 version          : 4.2.0
2020-03-09T18:43:44: PM2 log: Node.js version      : 12.13.1
2020-03-09T18:43:44: PM2 log: Current arch         : x64
2020-03-09T18:43:44: PM2 log: PM2 home             : /root/.pm2
2020-03-09T18:43:44: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2020-03-09T18:43:44: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2020-03-09T18:43:44: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2020-03-09T18:43:44: PM2 log: Application log path : /root/.pm2/logs
2020-03-09T18:43:44: PM2 log: Worker Interval      : 30000
2020-03-09T18:43:44: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2020-03-09T18:43:44: PM2 log: Concurrent actions   : 2
2020-03-09T18:43:44: PM2 log: SIGTERM timeout      : 1600
2020-03-09T18:43:44: PM2 log: ===============================================================================
2020-03-09T18:43:44: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2020-03-09T18:43:44: PM2 log: App [pm2-logrotate:0] online
2020-03-09T18:43:45: PM2 log: Stopping app:pm2-logrotate id:0
2020-03-09T18:43:45: PM2 error: Cancelling versioning data parsing
2020-03-09T18:43:45: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGINT]
2020-03-09T18:43:45: PM2 log: pid=23598 msg=process killed
2020-03-09T18:43:45: PM2 log: PM2 successfully stopped
2020-03-09T18:46:45: PM2 log: ===============================================================================
2020-03-09T18:46:45: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2020-03-09T18:46:45: PM2 log: Time                 : Mon Mar 09 2020 18:46:45 GMT+0700 (Western Indonesia Time)
2020-03-09T18:46:45: PM2 log: PM2 version          : 4.2.0
2020-03-09T18:46:45: PM2 log: Node.js version      : 12.13.1

The text was updated successfully, but these errors were encountered:

dgburton · 2020-04-22T21:28:23Z

I have this exact issue also...

garaboncias · 2020-04-23T07:12:13Z

We have same issue on pm2 startOrReload ecosystem.json --env production command which is stuck because somehow it starts a pm2 update in the middle of the reload process, but no one ask for it.

Unitech · 2020-04-23T11:26:10Z

Is it reproducing every time you run these commands?
Same Node.js version (12.13.1)? Same system (Debian)?

pm2-axon has not been updated for a while
https://github.com/Unitech/pm2-axon/blob/master/lib/sockets/pub.js#L47

garaboncias · 2020-04-23T12:31:20Z

Its happens approx once per five install. It started a week ago
We use node 10 but with the same Debian 10
We experience this with pm2 version 4.1, 4.2, 4.3
we use postDeploy.sh script and the same pm2-logrotate module...
As you see in our log, pm2 terminate and try resurrect,( but without success) and both process stuck

2020-04-21T11:16:56: PM2 log: ===============================================================================
2020-04-21T11:16:56: PM2 log: --- PM2 global error caught ---------------------------------------------------
2020-04-21T11:16:56: PM2 log: Time                 : Tue Apr 21 2020 11:16:56 GMT+0000 (Coordinated Universal Tim
e)
2020-04-21T11:16:56: PM2 error: Cannot read property 'writable' of undefined
2020-04-21T11:16:56: PM2 error: TypeError: Cannot read property 'writable' of undefined
    at PubSocket.send (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:47:14)
    at EventEmitter.<anonymous> (/usr/lib/node_modules/pm2/lib/Daemon.js:445:14)
    at EventEmitter.emit (/usr/lib/node_modules/pm2/node_modules/eventemitter2/lib/eventemitter2.js:330:22)
    at Console.console.(anonymous function) [as log] (/usr/lib/node_modules/pm2/lib/Utility.js:114:17)
    at Timeout._onTimeout (/usr/lib/node_modules/pm2/lib/God/Methods.js:168:15)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
2020-04-21T11:16:56: PM2 log: ===============================================================================
2020-04-21T11:16:56: PM2 error: [PM2] Resurrecting PM2

garaboncias · 2020-04-23T13:09:10Z

This is an other stackstrace:

/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:47
    if (sock.writable) sock.write(buf);
             ^

TypeError: Cannot read property 'writable' of undefined
    at PubSocket.send (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:47:14)
    at EventEmitter.<anonymous> (/usr/lib/node_modules/pm2/lib/Daemon.js:445:14)
    at EventEmitter.emit (/usr/lib/node_modules/pm2/node_modules/eventemitter2/lib/eventemitter2.js:330:22)
    at Object.God.notify (/usr/lib/node_modules/pm2/lib/Event.js:12:13)
    at Object.handleExit (/usr/lib/node_modules/pm2/lib/God.js:441:7)
    at Worker.cluExit (/usr/lib/node_modules/pm2/lib/God.js:276:13)
    at Object.onceWrapper (events.js:286:20)
    at Worker.emit (events.js:198:13)
    at Worker.EventEmitter.emit (domain.js:466:23)
    at ChildProcess.worker.process.once (internal/cluster/master.js:189:12)

garaboncias · 2020-05-22T05:50:49Z

Issue related to custom messaging among our instances through pm2 Api (pm2 bus, process.send).
I cannot reproduce on MacOs only in Debian.
here is a minimal demonstration:
https://drive.google.com/file/d/1cVL1-lDbqmJGX4BAzNWW6apLYXF0wUdE/view?usp=sharing
main.js:

const pm2 = require("pm2");
pm2.launchBus((err, bus) => {
    bus.on("process:123", packet => {
        console.log('got packet');
    });
});

worker.js:

setInterval(()=>{
    process.send({
        type: "process:123",
        data: {}
    });
});

ecosystem.json

{
  "apps": [
    {
      "name": "main",
      "script": "main.js",
      "exec_mode": "cluster_mode",
      "instances": 1
        },
    {
      "name": "worker",
      "script": "worker.js",
      "exec_mode": "cluster_mode",
      "instances": 6
    }
  ]
}

and you only need to run pm2 reload main command which always fail in my linux box

ghost · 2020-06-09T12:59:26Z

Issue appears quite easy to repro when concurrent requests are made for the same custom message.

2020-06-08T15:20:00: PM2 log: ===============================================================================
2020-06-08T15:20:00: PM2 log: --- PM2 global error caught ---------------------------------------------------
2020-06-08T15:20:00: PM2 log: Time                 : Mon Jun 08 2020 15:20:00 GMT-0400 (Eastern Daylight Time)
2020-06-08T15:20:00: PM2 error: Cannot read property 'writable' of undefined
2020-06-08T15:20:00: PM2 error: TypeError: Cannot read property 'writable' of undefined
    at PubSocket.send (/usr/local/nodejs-binary-12.16.3/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:47:14)
    at EventEmitter.<anonymous> (/usr/local/nodejs-binary-12.16.3/lib/node_modules/pm2/lib/Daemon.js:445:14)
    at EventEmitter.emit (/usr/local/nodejs-binary-12.16.3/lib/node_modules/pm2/node_modules/eventemitter2/lib/eventemitter2.js
:330:22)
    at Worker.cluMessage (/usr/local/nodejs-binary-12.16.3/lib/node_modules/pm2/lib/God/ClusterMode.js:65:24)
    at Worker.emit (events.js:322:22)
    at Worker.EventEmitter.emit (domain.js:505:15)
    at ChildProcess.<anonymous> (internal/cluster/worker.js:32:12)
    at ChildProcess.emit (events.js:310:20)
    at ChildProcess.EventEmitter.emit (domain.js:505:15)
    at emit (internal/child_process.js:876:12)
2020-06-08T15:20:00: PM2 log: ===============================================================================
2020-06-08T15:20:00: PM2 error: [PM2] Resurrecting PM2
2020-06-08T15:20:00: PM2 log: Stopping app:app id:0
2020-06-08T15:20:00: PM2 log: Stopping app:app id:1
...

PM2 version: 4.4.0
Node.js version: 12.16.3
OS: CentOS

aldipower · 2020-06-11T23:38:38Z

Same issue here. Happens suddenly and then all instances are down.

As @garaboncias we use pm2 messaging too, to collect metrics from our cluster workers. So, this is quite a lot of messaging. But not sure how this is related.

2020-06-11T22:40:00: PM2 log: ===============================================================================
2020-06-11T22:40:00: PM2 log: --- PM2 global error caught ---------------------------------------------------
2020-06-11T22:40:00: PM2 log: Time                 : Thu Jun 11 2020 22:40:00 GMT+0000 (Coordinated Universal Time)
2020-06-11T22:40:00: PM2 error: Cannot read property 'writable' of undefined
2020-06-11T22:40:00: PM2 error: TypeError: Cannot read property 'writable' of undefined
    at PubSocket.send (/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:47:14)
    at EventEmitter.<anonymous> (/usr/lib/node_modules/pm2/lib/Daemon.js:445:14)
    at EventEmitter.emit (/usr/lib/node_modules/pm2/node_modules/eventemitter2/lib/eventemitter2.js:330:22)
    at Worker.cluMessage (/usr/lib/node_modules/pm2/lib/God/ClusterMode.js:65:24)
    at Worker.emit (events.js:322:22)
    at Worker.EventEmitter.emit (domain.js:505:15)
    at ChildProcess.<anonymous> (internal/cluster/worker.js:32:12)
    at ChildProcess.emit (events.js:310:20)
    at ChildProcess.EventEmitter.emit (domain.js:505:15)
    at emit (internal/child_process.js:876:12)
2020-06-11T22:40:00: PM2 log: ===============================================================================
2020-06-11T22:40:00: PM2 error: [PM2] Resurrecting PM2
internal/modules/cjs/loader.js:983
  throw err;
  ^

Error: Cannot find module '/undefined'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:980:15)
    at Function.Module._load (internal/modules/cjs/loader.js:862:27)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)
    at internal/main/run_main_module.js:18:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Jun 11 22:40:00 pm2[22072]: [PM2] Spawning PM2 daemon with pm2_home=/home/felix/.pm2
Jun 11 22:40:00 pm2[22072]: [PM2] PM2 Successfully daemonized
Jun 11 22:40:00 pm2[22072]: [PM2] [v] Modules Stopped
Jun 11 22:40:00 pm2[22072]: [PM2][WARN] No process found
Jun 11 22:40:00 pm2[22072]: [PM2] [v] All Applications Stopped
Jun 11 22:40:00 pm2[22072]: [PM2] [v] PM2 Daemon Stopped

PM2 version: 4.4.0
Node.js version: 12.16.2
OS: Ubuntu 19.10

ChrisLahaye · 2020-07-07T11:25:18Z

@Unitech Wanted to share my observation: when to processes managed by pm2 call connect and launchBus, and one process calls disconnect, then the other process is affected by this with an error as result.

gotoin · 2020-08-04T08:42:03Z

me too

err: Cannot read property 'writable' of undefined

mod pm2-axon/lib/sockets/pub.js

PubSocket.prototype.send = function(msg){
  var socks = this.socks;
  var len = socks.length;
  var sock;

  var buf = this.pack(arguments);
  for (var i = 0; i < len; i++) {

    sock = socks[i];
    if (!sock) {
      continue
    }
    if (sock.writable) sock.write(buf);
  }

  return this;
};

I try just skip it...

gotoin · 2020-08-04T09:34:34Z

Unitech/pm2-axon#9

brettphillips05 · 2020-08-31T16:09:27Z

I started to receive the same error as mentioned above after I updated PM2. I am running the PM2 instance with the --no-daemon flag. Pulling in @gotoin change into global /pm2/node_modules/pm2-axon/lib/sockets/pub.js fixed the issue. Not ideal but glad the issue has been addressed, as I've spent multiple hours trying to fix this

govo · 2020-09-16T06:46:14Z

Unitech/pm2-axon#9

hope this can be update soon!

mattpr · 2022-11-29T10:12:40Z

We also get the pm2 crash reliably (pm2 5.2.2) running on ubuntu 20.04 LTS.

PM2 error: Cancelling versioning data parsing

When our server provisioning modifies the environment file for an app (e.g. to change app parameters or timeout configs) and asks pm2 to reload, that triggers the crash and causes an outage. Pretty awesome.

Unitech added the Inspecting label Apr 23, 2020

ghost mentioned this issue Jun 9, 2020

Cannot read property 'writable' of undefined ChrisLahaye/pm2-messages#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PM2 global error caught #4627

PM2 global error caught #4627

Snapper77 commented Mar 9, 2020

dgburton commented Apr 22, 2020

garaboncias commented Apr 23, 2020

Unitech commented Apr 23, 2020 •

edited

Loading

garaboncias commented Apr 23, 2020

garaboncias commented Apr 23, 2020

garaboncias commented May 22, 2020

ghost commented Jun 9, 2020 •

edited by ghost

Loading

aldipower commented Jun 11, 2020 •

edited

Loading

ChrisLahaye commented Jul 7, 2020

gotoin commented Aug 4, 2020 •

edited

Loading

gotoin commented Aug 4, 2020

brettphillips05 commented Aug 31, 2020

govo commented Sep 16, 2020

mattpr commented Nov 29, 2022

PM2 global error caught #4627

PM2 global error caught #4627

Comments

Snapper77 commented Mar 9, 2020

What's going wrong?

How could we reproduce this issue?

Supporting information

Log details

dgburton commented Apr 22, 2020

garaboncias commented Apr 23, 2020

Unitech commented Apr 23, 2020 • edited Loading

garaboncias commented Apr 23, 2020

garaboncias commented Apr 23, 2020

garaboncias commented May 22, 2020

ghost commented Jun 9, 2020 • edited by ghost Loading

aldipower commented Jun 11, 2020 • edited Loading

ChrisLahaye commented Jul 7, 2020

gotoin commented Aug 4, 2020 • edited Loading

gotoin commented Aug 4, 2020

brettphillips05 commented Aug 31, 2020

govo commented Sep 16, 2020

mattpr commented Nov 29, 2022

Unitech commented Apr 23, 2020 •

edited

Loading

ghost commented Jun 9, 2020 •

edited by ghost

Loading

aldipower commented Jun 11, 2020 •

edited

Loading

gotoin commented Aug 4, 2020 •

edited

Loading