PM2 Daemon is dead

[rahulkulmi]

I am using pm2 as a "pm2-docker" inside the docker container but it will stop response after some time and the last message i got from pm2 is "PM2 Daemon is dead" inside my log file.

I am not able to get pm2 logs because pm2 is inside the docker container. 
I am using latest pm2 version 2.7.2 and node version 7.10.1

When i am using pm2 2.6.1 that time i am not getting this problem so please tell me what is the issue and how i can fix this issue.

This pm2 issue is related to ec2 server memory or some thing else. please reply me ASAP

[johnwebbcole]

We were seeing this in our ECS container logs, and it turned out to be a hard memory limit in the task definition killing the container and PM2 Daemon is dead as a last gasp before ECS restarted the app.

If you don't have any health or monitoring on the container server, this probably isn't the issue.

[rahulkulmi]

@johnwebbcole Thanks for reply
ie. this issue is related to container memory like container didn't have memory for pm2, that is why pm2 daemon is dead or some thing else. Please let me know.

[DarkIsDude]

Hi, I have the same issue but I can't find log. @johnwebbcole , can you provide us more information? Thanks.

[johnwebbcole]

@rahulkulmi no, this was strictly an issue with our cloud formation health check killing the container on a hard memory limit. The PM2 Daemon is dead message was our only clue that something was killing the service from the outside the container.

If you don’t have anything monitoring your process, then this most likely isn’t the issue.

[rahulkulmi]

@johnwebbcole Sorry I am not getting your point properly but thanks for reply.

[vmarchaud]

@rahulkulmi He said that this log was printed because something send a SIGINT to the container

Closing since not related to PM2

[acuntex]

I've seen this error on various web apps, on one staging server (Ubuntu) and on AWS Linux.

Last night I also encountered it on a Debian system with a pm2-docker container with 2 simple javascripts that execute simple jobs. No indication at all why it was killed. The process uses about 2% of 10GB RAM so there is no memory problem.

Even /var/log/daemon.log showed no indication why docker should have killed it.
"docker ps -a" reports "Exited (1)"

[vmarchaud]

From the code of here, its looks like the PM2 Daemon took more than 4.5 seconds to repond so the CLI so it assumed that it crashed. Could you checkout in ~/.pm2/pm2.log inside the container if possible (with a volume bind to the host maybe) ?

[DarkIsDude]

@vmarchaud I can reproduce this issue every time. From my container pm2-docker start ecosystem.config.js return PM2 Daemon is dead but pm2 start ecosystem.config.js work fine, app start and I can see logs with pm2 logs.

• in /root/.pm2

2017-11-17 11:03:26: ===============================================================================
2017-11-17 11:03:26: --- New PM2 Daemon started ----------------------------------------------------
2017-11-17 11:03:26: Time                 : Fri Nov 17 2017 11:03:26 GMT+0100 (CET)
2017-11-17 11:03:26: PM2 version          : 2.7.2
2017-11-17 11:03:26: Node.js version      : 8.9.1
2017-11-17 11:03:26: Current arch         : x64
2017-11-17 11:03:26: PM2 home             : /PM2/
2017-11-17 11:03:26: PM2 PID file         : /PM2/pm2.pid
2017-11-17 11:03:26: RPC socket file      : /PM2/rpc.sock
2017-11-17 11:03:26: BUS socket file      : /PM2/pub.sock
2017-11-17 11:03:26: Application log path : /PM2/logs
2017-11-17 11:03:26: Process dump file    : /PM2/dump.pm2
2017-11-17 11:03:26: Concurrent actions   : 2
2017-11-17 11:03:26: SIGTERM timeout      : 1600
2017-11-17 11:03:26: ===============================================================================

Nothing in /root/.pm2/logs or /PM2/logs/

My Dockerfile

FROM node:8.9

LABEL MAINTAINER="technique@oveasoft.com"
LABEL version="2.0"

# Install package
RUN apt-get update
RUN apt-get -y install rsync

# Install npm package
RUN npm install -g pm2
RUN npm install -g istanbul

# Configure timezone
RUN echo "Europe/Paris" > /etc/timezone
RUN dpkg-reconfigure -f noninteractive tzdata

# Clean
RUN apt-get update && apt-get -y autoremove && apt-get clean

# Create folder for application
RUN mkdir /APP/
RUN mkdir /PM2/

# Configure ENV
ENV PM2_HOME /PM2/
ENV NODE_APP_INSTANCE 0

# Expose all
VOLUME ["/APP/"]
VOLUME ["/PM2/"]
EXPOSE 61337

WORKDIR /APP/

ENTRYPOINT ["/bin/bash", "-c", "setup/init.sh ${*}", "--"]

And I run it with docker run --name intra-scheduler -e NODE_ENV=dev -v /Users/doudou/Workspaces/OVEASOFT/intra-scheduler.git:/APP --entrypoint=/bin/bash -it intra-scheduler_local

[rkt2spc]

We're encountering this issue too, any updates on this?

[blimmer]

This is happening to me too. Turning off watch seems to fix the issue for me.

With this config

const dev = process.env.NODE_ENV === 'development';

let app = {
  // name of app in PM2 logs
  name: 'tmp-node-8-final',
  // scale app across CPUs available
  exec_mode: 'cluster',
  // as many instances as processor has cores
  instances: dev ? 1 : 0,
  // restart app on file change,
  watch: true,
  // files to ignore when watch is on
  ignore_watch: ['newrelic_agent.log', 'supervisord.log', '.git/*', '.idea/*', '.vscode/*'],
  // format of PM2 logs
  log_date_format: 'YYYY-MM-DD HH:mm:ss',
  // turn off PM2 logs within the container - they're forcing docker to run out of data space
  out_file: "/dev/stdout",
  error_file: "/dev/stderr",
  // number of consecutive unstable restarts before quit.
  max_restarts: 5,
};

// startup script
if (dev) {
  app.script = './src/dev-server.js';
} else {
  app.script = './build/server.js';
}

let config = {
  apps: [app],
};

module.exports = config;

it crashes every time

root@2967a5c6d44a:~/tmp-node-8-final# pm2-docker start pm2.config.js
0|tmp-node | 2017-12-12 23:36:22: CONFIG: missing version file /home/app/tmp-node-8-final/src/config/version.json
0|tmp-node | 2017-12-12 23:36:22: CONFIG: missing environment secret file /home/app/tmp-node-8-final/src/config/secret/development.json
0|tmp-node | 2017-12-12 23:36:26: Server is now running on http://localhost:3000
PM2 Daemon is dead

here's the pm2.log

2017-12-12 23:43:21: ===============================================================================
2017-12-12 23:43:21: --- New PM2 Daemon started ----------------------------------------------------
2017-12-12 23:43:21: Time                 : Tue Dec 12 2017 23:43:21 GMT+0000 (UTC)
2017-12-12 23:43:21: PM2 version          : 2.8.0
2017-12-12 23:43:21: Node.js version      : 8.9.1
2017-12-12 23:43:21: Current arch         : x64
2017-12-12 23:43:21: PM2 home             : /home/app/.pm2
2017-12-12 23:43:21: PM2 PID file         : /home/app/.pm2/pm2.pid
2017-12-12 23:43:21: RPC socket file      : /home/app/.pm2/rpc.sock
2017-12-12 23:43:21: BUS socket file      : /home/app/.pm2/pub.sock
2017-12-12 23:43:21: Application log path : /home/app/.pm2/logs
2017-12-12 23:43:21: Process dump file    : /home/app/.pm2/dump.pm2
2017-12-12 23:43:21: Concurrent actions   : 2
2017-12-12 23:43:21: SIGTERM timeout      : 1600
2017-12-12 23:43:21: ===============================================================================
2017-12-12 23:43:21: [Watch] Start watching tmp-node-8-final
2017-12-12 23:43:21: Starting execution sequence in -cluster mode- for app name:tmp-node-8-final id:0
2017-12-12 23:43:21: App name:tmp-node-8-final id:0 online
2017-12-12 23:43:21: (node:48) [DEP0007] DeprecationWarning: worker.suicide is deprecated. Please use worker.exitedAfterDisconnect.
2017-12-12 23:43:26: CONFIG: missing version file /home/app/tmp-node-8-final/src/config/version.json
2017-12-12 23:43:26: CONFIG: missing environment secret file /home/app/tmp-node-8-final/src/config/secret/development.json
2017-12-12 23:43:29: Server is now running on http://localhost:3000

when I change watch to false in the config, it does not crash.

[zontafil]

same issue here. Downgrading to 2.6.1 solves for me

[Unitech]

Fixed with new pm2-runtime:

RUN npm install Unitech/pm2#development -g
CMD [ 'pm2-runtime', 'app.js' ]

[rsilvestre]

@Unitech When do you think this fix will be merge on master branch. I'm not going to release a development version of pm2 in production.

[wallet77]

PM2 2.10.1 has been release:

$ npm install pm2@latest -g
$ pm2 update