PM2 v5.1.0: God Daemon taking up huge amounts of memory

[victor-ono]

#1126

I'm started experiencing this issue again after updating to 5.1.

It takes 230MB (~50%) of RAM on a 512MB machine

[yorickshan]

same issue, any updates？

[qiulang]

I have experienced the same problem with 5.2. The pm2 process takes up about 2.6G memory while the process it controls takes much lower memory 15 1 root S 2645m 34% 1 0% PM2 v5.2.0: God Daemon (/root/.pm2). It has lasted for more than 2 days till now.

The following is the output of pm2 report,

--- PM2 report ----------------------------------------------------------------
Date                 : Fri May 13 2022 14:33:40 GMT+0800 (China Standard Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.2.0
node version         : 14.18.1
node path            : not found
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : undefined
uid                  : 0
gid                  : 0
uptime               : 37450min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.2.0
node version         : 14.18.1
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : undefined
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz
cpus nb              : 4
freemem              : 227586048
totalmem             : 8201084928
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
┌─────┬──────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name             │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 1   │ ws_redis         │ default     │ 2.0.0   │ fork    │ 9154     │ 24D    │ 5    │ online    │ 0.3%     │ 49.8mb   │ root     │ disabled │
└─────┴──────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
Module
┌────┬──────────────────────────────┬───────────────┬──────────┬──────────┬──────┬──────────┬──────────┬──────────┐
│ id │ module                       │ version       │ pid      │ status   │ ↺    │ cpu      │ mem      │ user     │
├────┼──────────────────────────────┼───────────────┼──────────┼──────────┼──────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate                │ 2.7.0         │ 40       │ online   │ 1    │ 0.2%     │ 27.3mb   │ root     │
└────┴──────────────────────────────┴───────────────┴──────────┴──────────┴──────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        | 2022-04-19T10:34:07: PM2 log: Stopping app:ws_redis id:1
PM2        | 2022-04-19T10:34:07: PM2 log: App [ws_redis:1] exited with code [0] via signal [SIGINT]
PM2        | 2022-04-19T10:34:07: PM2 log: pid=31052 msg=process killed
PM2        | 2022-04-19T10:34:07: PM2 log: App [ws_redis:1] starting in -fork mode-
PM2        | 2022-04-19T10:34:07: PM2 log: App [ws_redis:1] online
PM2        | 2022-04-19T10:38:26: PM2 log: Stopping app:ws_redis id:1
PM2        | 2022-04-19T10:38:26: PM2 log: App [ws_redis:1] exited with code [0] via signal [SIGINT]
PM2        | 2022-04-19T10:38:26: PM2 log: pid=31208 msg=process killed
PM2        | 2022-04-19T10:38:26: PM2 log: App [ws_redis:1] starting in -fork mode-
PM2        | 2022-04-19T10:38:26: PM2 log: App [ws_redis:1] online
PM2        | 2022-04-19T10:52:09: PM2 log: Stopping app:ws_redis id:1
PM2        | 2022-04-19T10:52:09: PM2 log: App [ws_redis:1] exited with code [0] via signal [SIGINT]
PM2        | 2022-04-19T10:52:09: PM2 log: pid=31497 msg=process killed
PM2        | 2022-04-19T10:52:09: PM2 log: App [ws_redis:1] starting in -fork mode-
PM2        | 2022-04-19T10:52:09: PM2 log: App [ws_redis:1] online
PM2        | 2022-04-19T13:22:44: PM2 log: Stopping app:ws_redis id:1
PM2        | 2022-04-19T13:22:44: PM2 log: App [ws_redis:1] exited with code [0] via signal [SIGINT]
PM2        | 2022-04-19T13:22:44: PM2 log: pid=32384 msg=process killed
PM2        | 2022-04-19T13:22:44: PM2 log: App [ws_redis:1] starting in -fork mode-
PM2        | 2022-04-19T13:22:44: PM2 log: App [ws_redis:1] online

[qiulang]

The CPU usage is almost always less than 1%, that is why I suspect there is some memory leak somewhere

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
   15     1 root     S    2645m  34%   0   0% PM2 v5.2.0: God Daemon (/root/.pm2)
 9154    15 root     S     312m   4%   2   0% node /home/express/build/index.js
   40    15 root     S     279m   4%   0   0% node /root/.pm2/modules/pm2-logrotate/node_modules/pm2-log
 9175  8997 root     S     261m   3%   2   0% node /usr/bin/pm2 logs
32405 31926 root     T     261m   3%   2   0% node /usr/bin/pm2 logs
31584 31569 root     T     261m   3%   3   0% node /usr/bin/pm2 log 1
  497 31926 root     T     261m   3%   1   0% node /usr/bin/pm2 log 1
  639 31926 root     T     261m   3%   0   0% node /usr/bin/pm2 log 1
  953 31926 root     T     260m   3%   3   0% node /usr/bin/pm2 log 1

[TheAndroidGuy]

@qiulang This is not problem with PM2, changing the memory allocator to jemalloc will fix the memory issue.

[qiulang]

No I don't think so and nodejs/node#21973 was closed

[victor-ono]

I'm still having the same issue.

[sarojmhrzn]

I am facing the same issue - when I run the my node application with pm2 it takes huge amount of memory and crashes but when I simply just run "node app.js" then it run without any memory issue and crash.

[qiulang]

@sarojmhrzn which node version did you use? Now I use node 16.4 to see if this can fix the problem.

[EHyang]

i'm still having the same issue.
i used v5.2.0

[rodrigograca31]

same here. pm2 v5.2.0 and seems its not my code because when running with node file.js --inspect it stays at about 7mb memory looking at the heap snapshot, while with pm2 it says 47.2mb and keeps going up over time.

[rodrigograca31]

also I had 2 similar pieces of code and one didnt have thise memory leak problem....
then I noticed that one was using 4.x and updated it to 5.x.... and now in both the memory keeps increasing over time every time I run pm2 list

[smashah]

The daemon memory leak is insane, on a 256GB RAM VPS, pm2 is managing about 50 puppeteer + chrome isntances and yet pm2 Daemon is taking up 150GB+ of RAM!

HOW can I run the daemon with --inspect?? I've fixed memory leaks before but I just need to inspect the process in order to make some headway on this stupid memory leak.

pm2 report

--- PM2 report ----------------------------------------------------------------
Date                 : Fri Dec 16 2022 19:17:07 GMT+0800 (Singapore Standard Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.2.2
node version         : 18.12.1
node path            : /home/admin/orch/node_modules/.bin/pm2
argv                 : /usr/bin/node,/home/admin/orch/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : admin
uid                  : 1001
gid                  : 1001
uptime               : 4min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.2.2
node version         : 18.12.1
node path            : /home/admin/orch/node_modules/.bin/pm2
argv                 : /usr/bin/node,/home/admin/orch/node_modules/.bin/pm2,report
argv0                : node
user                 : admin
uid                  : 1001
gid                  : 1001
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : AMD EPYC 7282 16-Core Processor
cpus nb              : 64
freemem              : 242047713280
totalmem             : 270254182400
home                 : /home/admin
===============================================================================

Investigation

Update 1:

This is how you can turn on inspect on the daemon:

> PM2_NODE_OPTIONS='--inspect' pm2 update

Update 2:

From initial inspection, it looks like the fact that pidusage can take some time (few seconds sometimes) results in the callback(?) hanging around and hogging a lot of memory? Or the result/buffer of pidusage is leaking somewhere?

Reference:

soyuka/pidusage@ff04d9e
soyuka/pidusage#17

Update 3:

After hooking into the process on the machine with the issue, it looks like the heap is fine but rss continually increases. Apparently this is an issue with glib malloc. After switching to jemalloc the daemon RAM issue has been alleviated. But now we're hitting a new ENFILE issue:

2022-12-17T17:35:27: PM2 error: Trace: [Error: ENFILE: file table overflow, open '/home/admin/.pm2/logs/api-639abec721066650a072aded-1-out.log'] {
  errno: -23,
  code: 'ENFILE',
  syscall: 'open',
  path: '/home/admin/.pm2/logs/api-639abec721066650a072aded-1-out.log'
}
    at God.logAndGenerateError (/home/admin/orch/node_modules/pm2/lib/God/Methods.js:34:15)
    at /home/admin/orch/node_modules/pm2/lib/God/ForkMode.js:87:13
    at wrapper (/home/admin/orch/node_modules/async/internal/once.js:12:16)
    at WriteStream.next (/home/admin/orch/node_modules/async/waterfall.js:96:20)
    at WriteStream.<anonymous> (/home/admin/orch/node_modules/async/internal/onlyOnce.js:12:16)
    at Object.onceWrapper (node:events:628:26)
    at WriteStream.emit (node:events:513:28)
    at WriteStream.emit (node:domain:489:12)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)

Update 4:

The ENFILE issue was directly to me adding pidusage.clear() to the end of the pidusage callbacks. This would cause too many accesses to ps or proc files resulting in the ENFILE error. I reverted that change and the ENFILE error disappeared completely.

The solution to this issue is to let jemalloc to manage the memory for the daemon process. At first I set jemalloc as the default globally but it caused a regression on my processes managed by pm2 (puppeteer/puppeteer#8246 (comment)). So make sure you only run the Daemon with jemalloc to avoid regressions in your orchestrated processes.

# as root
> apt-get install libjemalloc-dev

Then to start your pm2 daemon with jemalloc then run:

> LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so pm2 update

You can then confirm that pm2 daemon is running with jemalloc by using:

> ps -aux | grep pm2 #Use this to find the PID of the daemon
> cat /proc/PM2_DAEMON_PID/smaps | grep jemalloc

Before the daemon would balloon and take all available memory in the server (256GB RAM) and eventually crash. Now daemon is hovering around 150-300MB of RAM.

[smashah]

tldr: Before the daemon would balloon and take all available memory in the server (256GB RAM) and eventually crash. Now daemon is hovering around 150-300MB of RAM after forcing the daemon to use jemalloc.

@rodrigograca31 can you check the above solution and report back if it works for you please. Maybe then I will make a PR to automatically detect and use jemalloc for the Daemon.

[rodrigograca31]

I can't anymore... it fixed it self over time if you know what I mean 😂

[prithvisharma]

@smashah how do i install jemalloc on amazon linux 2? facing the same memory spike issue

[smashah]

@smashah how do i install jemalloc on amazon linux 2? facing the same memory spike issue

Try this
sudo yum install -y jemalloc-devel

Then use

npx jemalloc-check to find the path for jemalloc.

[victor-ono]

I stopped using PM2. Instead I'm using Linux native systemd without any memory or other issues.

[titanism]

Indeed we determined that pmx (e.g. via @pm2/io) is the culprit for memory leaks. Typically our app runs at 500MB but without explicitly disabling pmx, the app grew to over 3-4GB in memory.

To fix this, simply disable pmx in your ecosystem.json file:

{
  "apps": [
    {
      "name": "your-app-name",
      "script": "app.js",
      "exec_mode": "cluster_mode",
      "wait_ready": true,
      "instances": "max",
+      "pmx": false,
      "env_production": {
        "NODE_ENV": "production"
      }
    },

Then delete and restart all via pm2 delete all and pm2 start ecosystem.json --env production.

References:

• pm2/lib/ProcessUtils.js

  Lines 5 to 6 in 2573516

  	if (process.env.pmx !== 'false') {
  	const pmx = require('@pm2/io')

• Memory leak #4510 (comment)
• forwardemail/forwardemail.net@748b213

[titanism]

Possibly related #5216

[vicatcu]

I'm running PM2 5.3.0 with Node 18.15.0 on a Raspberry Pi... and top shows

812 vic       20   0  188412  71332  33324 R 100.7   3.8   2:56.91 PM2 v5.3.0: God

After a little while, it the memory utilization takes over all available resources and the system crashes. How exactly do I fix this? It's not clear to me from reviewing this thread. I don't know what ecosystem.json is, does it apply to my configuration?

[titanism]

@vicatcu set pmx: false option, it's clearly described here #5145 (comment). Yes if you're using PM2 then you should use an ecosystem.json file as its configuration. Please see PM2 docs for how to use this.

[vicatcu]

Thanks I've been using PM2 for a long time and ecosystem.json is new to me... I usually just do pm2 start run.js --name somename and then do pm2 save and then pm2 startup when everything is running the way I want it to. So my followup questions are:

1. Is there a solution for people that use pm2 like me, without an ecosystem.json file, or
2. Is there a 'migration guide' to how to convert a situation like what I described to using ecosystem.json instead?

[rachitbucha]

Any update on this issue

[titanism]

Just set pmx: false in the interim, it doesn't seem like PM2 is as actively maintained as it used to be.

[tintobrazov]

The problem was the incompatibility of the Node.js and PM2 versions.

Was
node.js 21.6.1
pm2 5.3.1

It became
node.js 18.9.0
pm2 5.3.1

And everything worked as it should

[zyhnbyyds]

The problem was the incompatibility of the Node.js and PM2 versions.

Was node.js 21.6.1 pm2 5.3.1

It became node.js 18.9.0 pm2 5.3.1

And everything worked as it should

it works

[Mikkou]

pmx didn't help

config
module.exports = {
apps: [
{
name: 'app-3160',
script: 'dist/src/main.js',
exec_mode: 'cluster',
instances: 2,
watch: false,
max_memory_restart: '4G',
listen_timeout: 22000,
kill_timeout: 8000,
restart_delay: 4000,
time: true,
cwd: '.',
wait_ready: true,
instance_var: 'APP_INSTANCE_SEQ',
pmx: false,
env: {
ENV: 'prod',
NODE_ENV: 'production',
NODE_OPTIONS: '--max-old-space-size=4096',
},
},
],
};