If PM2 is killed, the dump.pm2 file is empty #2827

extensionsapp · 2017-04-10T20:31:55Z

Hello.

~# pm2 start process.json && pm2 save

process.json

{
  "apps" : [{
    "name" : "zal.su",
    "script" : "/home/zal.su/app.js",
    "cwd" : "/home/zal.su",
    "error_file" : "/home/zal.su/log/err.log",
    "out_file" : "/home/zal.su/log/out.log",
    "node_args" : "--max-old-space-size=768",
    "max_memory_restart" : "768M",
    "watch" : true,
    "ignore_watch" : [
      "config",
      "log",
      "node_modules",
      "themes/*/public",
      "themes/.*",
      ".*"
    ],
    "watch_options" : {
      "followSymlinks": false
    },
    "vizion" : false
  }]
}

My log:

2017-04-10 19:30:25: pm2 has been killed by signal, dumping process list before exit...
2017-04-10 19:30:26: Deleting process 1
2017-04-10 19:30:26: Stopping app:zal.su id:1
2017-04-10 19:30:26: App [pm2-logrotate] with id [2] and pid [3817], exited with code [0] via signal [SIGTERM]
2017-04-10 19:30:26: App [zal.su] with id [1] and pid [11334], exited with code [0] via signal [SIGTERM]
2017-04-10 19:30:26: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:2
2017-04-10 19:30:26: App name:pm2-logrotate id:2 online
2017-04-10 19:30:26: pid=11334 msg=process killed
2017-04-10 19:30:26: Deleting process 2
2017-04-10 19:30:26: Stopping app:pm2-logrotate id:2
2017-04-10 19:30:26: App [pm2-logrotate] with id [2] and pid [12030], exited with code [0] via signal [SIGINT]
2017-04-10 19:30:27: pid=12030 msg=process killed
2017-04-10 19:30:27: [PM2] Exited peacefully

I reboot server, and my App zal.su not start.

2017-04-10 22:44:31: ===============================================================================
2017-04-10 22:44:31: --- New PM2 Daemon started ----------------------------------------------------
2017-04-10 22:44:31: Time                 : Mon Apr 10 2017 22:44:31 GMT+0300 (MSK)
2017-04-10 22:44:31: PM2 version          : 2.4.2
2017-04-10 22:44:31: Node.js version      : 6.10.2
2017-04-10 22:44:31: Current arch         : x64
2017-04-10 22:44:31: PM2 home             : /root/.pm2
2017-04-10 22:44:31: PM2 PID file         : /root/.pm2/pm2.pid
2017-04-10 22:44:31: RPC socket file      : /root/.pm2/rpc.sock
2017-04-10 22:44:31: BUS socket file      : /root/.pm2/pub.sock
2017-04-10 22:44:31: Application log path : /root/.pm2/logs
2017-04-10 22:44:31: Process dump file    : /root/.pm2/dump.pm2
2017-04-10 22:44:31: Concurrent actions   : 2
2017-04-10 22:44:31: SIGTERM timeout      : 1600
2017-04-10 22:44:31: ===============================================================================
2017-04-10 22:44:32: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-04-10 22:44:32: App name:pm2-logrotate id:0 online

Why did not the application start after reboot?

The text was updated successfully, but these errors were encountered:

vmarchaud · 2017-04-20T09:11:25Z

Can you share the service unit file (/etc/systemd/system/pm2.service with systemd) ?

extensionsapp · 2017-04-20T09:37:29Z

cat /etc/systemd/system/pm2-root.service

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid

ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Unitech · 2017-04-21T14:39:09Z

The dump does not exists after running pm2 start process.json && pm2 save?

After running this command can you run ll ~/.pm2/dump.pm2?

extensionsapp · 2017-04-26T15:41:08Z

# pm2 start process.json && pm2 save
[PM2][WARN] Applications menopo.space not running, starting...
[PM2] App [menopo.space] launched (1 instances)
┌──────────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐
│ App name     │ id │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem       │ watching │
├──────────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤
│ menopo.space │ 1  │ fork │ 1378 │ online │ 0       │ 0s     │ 50% │ 20.5 MB   │ enabled  │
└──────────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 33.324 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app
[PM2] Saving current process list...
[PM2] Successfully saved in /root/.pm2/dump.pm2
# ls -l ~/.pm2/dump.pm2
-rw-r--r-- 1 root root 3189 Apr 26 18:28 /root/.pm2/dump.pm2
# pm2 logs
-bash: fork: Cannot allocate memory
# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬──────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu  │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼──────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 100% │ 31.711 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴──────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app

after reboot, my app not start.

# cat .pm2/dump.pm2
[]
# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 32.301 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app

extensionsapp · 2017-05-01T11:08:58Z

<--- Last few GCs --->

1207491733 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.5 / 0.0 ms [allocation failure].
1207492861 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.6 / 0.0 ms [allocation failure].
1207497444 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.5 / 0.0 ms [allocation failure].
1207500246 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.6 / 0.0 ms [allocation failure].
1207505179 ms: Scavenge 17.2 (24.0) -> 16.2 (24.0) MB, 0.6 / 0.0 ms [allocation failure].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x20aa14ecfb39 <JS Object>
    1: stringify [native json.js:178] [pc=0x275ddf61c9b7] (this=0x20aa14ec90f9 <a JSON with map 0x1dd74670a0e1>,E=0x33c5ad3ff401 <an Object with map 0x390cd35a4ef9>,F=0x20aa14e04381 <undefined>,S=0x20aa14e04381 <undefined>)
    2: arguments adaptor frame: 1->3
    3: send [/usr/lib/node_modules/pm2/node_modules/pm2-axon/lib/sockets/pub.js:~38] [pc=0x275ddfb181aa] (this=0x14677e13c649 <a PubSock...

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 2: 0x10c9a4c [PM2 v2.4.4: God Daemon (/root/.pm2)]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 7: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 8: v8::internal::IncrementalStringBuilder::Extend() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 9: v8::internal::BasicJsonStringifier::SerializeString(v8::internal::Handle<v8::internal::String>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
10: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
11: v8::internal::BasicJsonStringifier::Result v8::internal::BasicJsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
12: v8::internal::BasicJsonStringifier::Stringify(v8::internal::Handle<v8::internal::Object>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
13: v8::internal::Runtime_BasicJSONStringify(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
14: 0x275dde8092a7
2017-05-01 11:50:37: ===============================================================================
2017-05-01 11:50:37: --- New PM2 Daemon started ----------------------------------------------------
2017-05-01 11:50:37: Time                 : Mon May 01 2017 11:50:37 GMT+0300 (MSK)
2017-05-01 11:50:37: PM2 version          : 2.4.4
2017-05-01 11:50:37: Node.js version      : 6.10.2
2017-05-01 11:50:37: Current arch         : x64
2017-05-01 11:50:37: PM2 home             : /root/.pm2
2017-05-01 11:50:37: PM2 PID file         : /root/.pm2/pm2.pid
2017-05-01 11:50:37: RPC socket file      : /root/.pm2/rpc.sock
2017-05-01 11:50:37: BUS socket file      : /root/.pm2/pub.sock
2017-05-01 11:50:37: Application log path : /root/.pm2/logs
2017-05-01 11:50:37: Process dump file    : /root/.pm2/dump.pm2
2017-05-01 11:50:37: Concurrent actions   : 2
2017-05-01 11:50:37: SIGTERM timeout      : 1600
2017-05-01 11:50:37: ===============================================================================
2017-05-01 11:50:38: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-05-01 11:50:38: App name:pm2-logrotate id:0 online
2017-05-01 11:50:38: Stopping app:pm2-logrotate id:0
2017-05-01 11:50:38: App [pm2-logrotate] with id [0] and pid [27004], exited with code [0] via signal [SIGINT]
2017-05-01 11:50:38: [HandleExit] PM2 is being killed, stopping restart procedure...
2017-05-01 11:50:38: pid=27004 msg=process killed
2017-05-01 11:50:38: ===============================================================================
2017-05-01 11:50:38: --- Stopping PM2 --------------------------------------------------------------
2017-05-01 11:50:38: Time                 : Mon May 01 2017 11:50:38 GMT+0300 (MSK)
2017-05-01 11:50:38: ===============================================================================
2017-05-01 11:50:38: RPC closed
2017-05-01 11:50:38: PUB closed
2017-05-01 11:50:38: PM2 successfully stopped
2017-05-01 13:46:00: ===============================================================================
2017-05-01 13:46:00: --- New PM2 Daemon started ----------------------------------------------------
2017-05-01 13:46:00: Time                 : Mon May 01 2017 13:46:00 GMT+0300 (MSK)
2017-05-01 13:46:00: PM2 version          : 2.4.4
2017-05-01 13:46:00: Node.js version      : 6.10.2
2017-05-01 13:46:00: Current arch         : x64
2017-05-01 13:46:00: PM2 home             : /root/.pm2
2017-05-01 13:46:00: PM2 PID file         : /root/.pm2/pm2.pid
2017-05-01 13:46:00: RPC socket file      : /root/.pm2/rpc.sock
2017-05-01 13:46:00: BUS socket file      : /root/.pm2/pub.sock
2017-05-01 13:46:00: Application log path : /root/.pm2/logs
2017-05-01 13:46:00: Process dump file    : /root/.pm2/dump.pm2
2017-05-01 13:46:00: Concurrent actions   : 2
2017-05-01 13:46:00: SIGTERM timeout      : 1600
2017-05-01 13:46:00: ===============================================================================
2017-05-01 13:46:00: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-05-01 13:46:00: App name:pm2-logrotate id:0 online

cat /root/.pm2/dump.pm2
[...

...]

But if I reboot dump.pm2 empty!

cat /root/.pm2/dump.pm2
[]

pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 32.777 MB   │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┘
 Use `pm2 show <id|name>` to get more details about an app

I need websites to be always online! Please do something.

Unitech · 2017-05-01T11:19:46Z

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory

extensionsapp · 2017-05-01T11:36:10Z

@Unitech There was a splash in the load, memory refused. But why PM2 does not restart the processes? And /root/.pm2/dump.pm2 - empty []?

extensionsapp · 2017-06-12T14:03:11Z

I have error ENOMEM - OK;
All process stop - OK;
I read file /root/.pm2/dump.pm2 - OK;
I reboot server - OK;
I read file /root/.pm2/dump.pm2 - EMPTY FILE [], Why?

2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:31: --- PM2 global error caught --------------------------------------------                                                                                                -------
2017-06-12 11:50:31: Time                 : Mon Jun 12 2017 11:50:31 GMT+0300 (MSK)
2017-06-12 11:50:31: spawn ENOMEM
2017-06-12 11:50:31: Error: spawn ENOMEM
    at exports._errnoException (util.js:1018:11)
    at ChildProcess.spawn (internal/child_process.js:319:11)
    at exports.spawn (child_process.js:378:9)
    at /usr/lib/node_modules/pm2/lib/TreeKill.js:33:14
    at buildProcessTree (/usr/lib/node_modules/pm2/lib/TreeKill.js:79:12)
    at module.exports (/usr/lib/node_modules/pm2/lib/TreeKill.js:32:5)
    at Object.God.killProcess (/usr/lib/node_modules/pm2/lib/God/Methods.js:223:7)
    at Object.God.stopProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:265:9)
    at Object.God.restartProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:362:11                                                                                                )
    at /usr/lib/node_modules/pm2/lib/God/ActionMethods.js:395:20
2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:31: [PM2] Resurrecting PM2
internal/child_process.js:319
    throw errnoException(err, 'spawn');
    ^

Error: spawn ENOMEM
    at exports._errnoException (util.js:1018:11)
    at ChildProcess.spawn (internal/child_process.js:319:11)
    at exports.spawn (child_process.js:378:9)
    at /usr/lib/node_modules/pm2/lib/TreeKill.js:33:14
    at buildProcessTree (/usr/lib/node_modules/pm2/lib/TreeKill.js:79:12)
    at module.exports (/usr/lib/node_modules/pm2/lib/TreeKill.js:32:5)
    at Object.God.killProcess (/usr/lib/node_modules/pm2/lib/God/Methods.js:223:7)
    at Object.God.stopProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:265:9)
    at Object.God.restartProcessId (/usr/lib/node_modules/pm2/lib/God/ActionMethods.js:362:11                                                                                                )
    at /usr/lib/node_modules/pm2/lib/God/ActionMethods.js:395:20
2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:31: --- New PM2 Daemon started ---------------------------------------------                                                                                                -------
2017-06-12 11:50:31: Time                 : Mon Jun 12 2017 11:50:31 GMT+0300 (MSK)
2017-06-12 11:50:31: PM2 version          : 2.4.4
2017-06-12 11:50:31: Node.js version      : 6.10.2
2017-06-12 11:50:31: Current arch         : x64
2017-06-12 11:50:31: PM2 home             : /root/.pm2
2017-06-12 11:50:31: PM2 PID file         : /root/.pm2/pm2.pid
2017-06-12 11:50:31: RPC socket file      : /root/.pm2/rpc.sock
2017-06-12 11:50:31: BUS socket file      : /root/.pm2/pub.sock
2017-06-12 11:50:31: Application log path : /root/.pm2/logs
2017-06-12 11:50:31: Process dump file    : /root/.pm2/dump.pm2
2017-06-12 11:50:31: Concurrent actions   : 2
2017-06-12 11:50:31: SIGTERM timeout      : 1600
2017-06-12 11:50:31: ========================================================================                                                                                                =======
2017-06-12 11:50:32: Starting execution sequence in -fork mode- for app name:pm2-logrotate id                                                                                                :0
2017-06-12 11:50:32: App name:pm2-logrotate id:0 online
2017-06-12 11:50:32: Stopping app:pm2-logrotate id:0
2017-06-12 11:50:32: App [pm2-logrotate] with id [0] and pid [19674], exited with code [0] vi                                                                                                a signal [SIGINT]
2017-06-12 11:50:32: [HandleExit] PM2 is being killed, stopping restart procedure...
2017-06-12 11:50:32: pid=19674 msg=process killed
2017-06-12 11:50:32: ========================================================================                                                                                                =======
2017-06-12 11:50:32: --- Stopping PM2 -------------------------------------------------------                                                                                                -------
2017-06-12 11:50:32: Time                 : Mon Jun 12 2017 11:50:32 GMT+0300 (MSK)
2017-06-12 11:50:32: ========================================================================                                                                                                =======
2017-06-12 11:50:32: RPC closed
2017-06-12 11:50:32: PUB closed
2017-06-12 11:50:32: PM2 successfully stopped

Please, tell me why after the reboot the /root/.pm2/dump.pm2 file becomes empty []?

After reboot I have memory

~# free
             total       used       free     shared    buffers     cached
Mem:       1024416     237764     786652      14168      19252     104068
-/+ buffers/cache:     114444     909972
Swap:            0          0          0

but app`s not start.

extensionsapp · 2017-06-12T15:50:33Z

This is probably the biggest problem with PM2, since I can not be sure that my sites will have 99.99% uptime.

extensionsapp · 2017-07-30T21:09:38Z

2017-07-31 00:00:14: Change detected on path get.js for app test.website.org - restarting
2017-07-31 00:00:14: Stopping app:test.website.org id:12
2017-07-31 00:00:14: App [test.website.org] with id [12] and pid [6444], exited with code [0] via signal [SIGINT]
2017-07-31 00:00:14: pid=6444 msg=process killed
2017-07-31 00:00:14: Starting execution sequence in -fork mode- for app name:test.website.org id:12
2017-07-31 00:00:14: App name:test.website.org id:12 online

<--- Last few GCs --->

4172772419 ms: Scavenge 20.7 (27.0) -> 19.7 (27.0) MB, 1.1 / 0.0 ms [allocation failure].
4172787059 ms: Scavenge 20.7 (27.0) -> 19.7 (27.0) MB, 0.5 / 0.0 ms [allocation failure].
4172800147 ms: Scavenge 20.6 (27.0) -> 19.7 (27.0) MB, 0.9 / 0.0 ms [allocation failure].
4172813613 ms: Scavenge 20.7 (27.0) -> 19.7 (27.0) MB, 1.1 / 0.0 ms [allocation failure].
4172814008 ms: Scavenge 20.7 (27.0) -> 19.8 (27.0) MB, 1.5 / 0.0 ms [allocation failure].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1520f60cfb39 <JS Object>
    1: parse [native json.js:~61] [pc=0x3bd85db7ad21] (this=0x1520f60c90f9 <a JSON with map 0x2037fb90a0e1>,C=0x94d94dfff59 <String[91]: {"type":"axm:monitor","data":{"Loop delay":{"value":"0.52ms","agg_type":"avg","alert":{}}}}>,w=0x1520f6004381 <undefined>)
    2: arguments adaptor frame: 1->2
    3: onread [internal/child_process.js:457] [pc=0x3bd85db3880e] (this=0x24bedbc649d1 <a Pipe with ...

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 2: 0x10c9a4c [PM2 v2.4.4: God Daemon (/root/.pm2)]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 6: v8::internal::Heap::CollectGarbage(v8::internal::GarbageCollector, char const*, char const*, v8::GCCallbackFlags) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 7: v8::internal::Factory::NewRawOneByteString(int, v8::internal::PretenureFlag) [PM2 v2.4.4: God Daemon (/root/.pm2)]
 8: v8::internal::Handle<v8::internal::String> v8::internal::JsonParser<false>::ScanJsonString<true>() [PM2 v2.4.4: God Daemon (/root/.pm2)]
 9: v8::internal::JsonParser<false>::ParseJsonObject() [PM2 v2.4.4: God Daemon (/root/.pm2)]
10: v8::internal::JsonParser<false>::ParseJsonValue() [PM2 v2.4.4: God Daemon (/root/.pm2)]
11: v8::internal::JsonParser<false>::ParseJson() [PM2 v2.4.4: God Daemon (/root/.pm2)]
12: v8::internal::JsonParser<false>::Parse(v8::internal::Handle<v8::internal::String>) [PM2 v2.4.4: God Daemon (/root/.pm2)]
13: v8::internal::Runtime_ParseJson(int, v8::internal::Object**, v8::internal::Isolate*) [PM2 v2.4.4: God Daemon (/root/.pm2)]
14: 0x3bd85d4092a7
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: --- New PM2 Daemon started ----------------------------------------------------
2017-07-31 00:00:19: Time                 : Mon Jul 31 2017 00:00:19 GMT+0300 (MSK)
2017-07-31 00:00:19: PM2 version          : 2.4.4
2017-07-31 00:00:19: Node.js version      : 6.10.2
2017-07-31 00:00:19: Current arch         : x64
2017-07-31 00:00:19: PM2 home             : /root/.pm2
2017-07-31 00:00:19: PM2 PID file         : /root/.pm2/pm2.pid
2017-07-31 00:00:19: RPC socket file      : /root/.pm2/rpc.sock
2017-07-31 00:00:19: BUS socket file      : /root/.pm2/pub.sock
2017-07-31 00:00:19: Application log path : /root/.pm2/logs
2017-07-31 00:00:19: Process dump file    : /root/.pm2/dump.pm2
2017-07-31 00:00:19: Concurrent actions   : 2
2017-07-31 00:00:19: SIGTERM timeout      : 1600
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-07-31 00:00:19: App name:pm2-logrotate id:0 online
2017-07-31 00:00:19: Stopping app:pm2-logrotate id:0
2017-07-31 00:00:19: App [pm2-logrotate] with id [0] and pid [6545], exited with code [0] via signal [SIGINT]
2017-07-31 00:00:19: [HandleExit] PM2 is being killed, stopping restart procedure...
2017-07-31 00:00:19: pid=6545 msg=process killed
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: --- Stopping PM2 --------------------------------------------------------------
2017-07-31 00:00:19: Time                 : Mon Jul 31 2017 00:00:19 GMT+0300 (MSK)
2017-07-31 00:00:19: ===============================================================================
2017-07-31 00:00:19: RPC closed
2017-07-31 00:00:19: PUB closed
2017-07-31 00:00:19: PM2 successfully stopped
2017-07-31 00:01:43: ===============================================================================
2017-07-31 00:01:43: --- New PM2 Daemon started ----------------------------------------------------
2017-07-31 00:01:43: Time                 : Mon Jul 31 2017 00:01:43 GMT+0300 (MSK)
2017-07-31 00:01:43: PM2 version          : 2.4.4
2017-07-31 00:01:43: Node.js version      : 6.10.2
2017-07-31 00:01:43: Current arch         : x64
2017-07-31 00:01:43: PM2 home             : /root/.pm2
2017-07-31 00:01:43: PM2 PID file         : /root/.pm2/pm2.pid
2017-07-31 00:01:43: RPC socket file      : /root/.pm2/rpc.sock
2017-07-31 00:01:43: BUS socket file      : /root/.pm2/pub.sock
2017-07-31 00:01:43: Application log path : /root/.pm2/logs
2017-07-31 00:01:43: Process dump file    : /root/.pm2/dump.pm2
2017-07-31 00:01:43: Concurrent actions   : 2
2017-07-31 00:01:43: SIGTERM timeout      : 1600
2017-07-31 00:01:43: ===============================================================================
2017-07-31 00:01:43: Starting execution sequence in -fork mode- for app name:pm2-logrotate id:0
2017-07-31 00:01:43: App name:pm2-logrotate id:0 online

Add automatic start sites after the fall.

extensionsapp · 2017-09-24T19:34:54Z

Please correct one single problem.

!!!Do not clear the dump file after rebooting the server!!!

If I have error "Error: spawn ENOMEM", my dump file:

~# cat /root/.pm2/dump.pm2
[ {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
},   {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
}
...
]

After reboot:

~# cat /root/.pm2/dump.pm2
[]

Why is it empty []?

extensionsapp · 2017-09-26T12:40:17Z

@Unitech, @vmarchaud please, read my message, step-by-step.

Start all process and pm2 save;
In a few hours ERROR: spawn ENOMEM or process out of memory;

~# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┬──────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │ user │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┼──────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 37.957 MB   │ root │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┴──────┘
 Use `pm2 show <id|name>` to get more details about an app

~# cat /root/.pm2/dump.pm2
[ {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
},   {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
}
...
]

cp /root/.pm2/dump.pm2 /root/.pm2/dump2.pm2
reboot

~# pm2 list
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
 Module activated
┌───────────────┬─────────┬────────────┬────────┬─────────┬─────┬─────────────┬──────┐
│ Module        │ version │ target PID │ status │ restart │ cpu │ memory      │ user │
├───────────────┼─────────┼────────────┼────────┼─────────┼─────┼─────────────┼──────┤
│ pm2-logrotate │ 2.2.0   │ N/A        │ online │ 0       │ 0%  │ 37.957 MB   │ root │
└───────────────┴─────────┴────────────┴────────┴─────────┴─────┴─────────────┴──────┘
 Use `pm2 show <id|name>` to get more details about an app

~# cat /root/.pm2/dump.pm2
[]

dump2 > dump. cp /root/.pm2/dump2.pm2 /root/.pm2/dump.pm2
resurrect. pm2 resurrect

[PM2] Resurrecting
[PM2] Restoring processes located in /root/.pm2/dump.pm2
[PM2] Process /test1/app.js restored
[PM2] Process /test2/app.js restored
[PM2] Process /test3/app.js restored
[PM2] Process /test4/app.js restored
[PM2] Process /test5/app.js restored
[PM2] Process /test6/app.js restored
[PM2] Process /test7/app.js restored
[PM2] Process /test8/app.js restored
[PM2] Process /test9/app.js restored
[PM2] Process /test10/app.js restored
...
NICE

~# cat /root/.pm2/dump.pm2
[ {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
},   {
    "instance_var": "NODE_APP_INSTANCE",
    "exec_mode": "fork_mode",
    "env": {
    ...
}
...
]

Why is it empty [] dump after reboot?

radkodinev · 2017-10-22T17:36:08Z

Steps to reproduce:

systemctl start pm2-node.service
pm2 start someapp
pm2 save

Now dump.pm2 is okay.
Then kill pm2:

sudo kill -9 $(pgrep PM2)

Now dump.pm2 is empty and the process list is lost, no way to resurrect.
That's insane and totally makes pm2 unreliable!

denisenkoaj · 2018-02-21T21:23:02Z

@radkodinev @extensionsapp Hello guys! Did you solve problem ? i've shocked about exist it and i faced it with it too.

denisenkoaj · 2018-02-21T21:27:13Z

I've tried to set flag +i for dump file, it is solve problem , but i have next - i use keymetrics monitoring service, and if file have +i flag - keymetrics cannot write to it and cannot monitoring my server. It's problem if you use this service.

Unitech · 2018-03-02T14:50:51Z

will be fixed via: #3510

Unitech · 2018-04-09T09:53:48Z

If you want to get the fix now you can use the dev branch:

$ npm install Unitech/pm2#development -g

shixukai · 2018-04-20T12:20:40Z

@Unitech

If you want to get the fix now you can use the dev branch:

$ npm install Unitech/pm2#development -g

it seems the problem is still here.

jazarja · 2018-05-18T14:13:56Z

I also found this issue.

dreaganluna · 2018-07-02T12:35:59Z

We're also experiencing this. Is this planned to be added soon? Because this is a major bug imo.

radkodinev · 2018-07-02T12:47:40Z

Indeed, it is a major bug IMO as well. I cannot believe so much time has passed and it still hasn't been resolved. I strongly dislike products that have many fancy features but cannot get the basics right. I really hope this gets fixed very soon.

denisenkoaj · 2018-07-03T02:48:51Z

I've stoped paying to keymetrics due this bug, after i wrote them on mail about it, they wrote comment

If you want to get the fix now you can use the dev branch:
npm install Unitech/pm2#development -g

but it's yet here, and i think it's really major bug

wallet77 · 2018-07-04T12:34:07Z

PM2 3.0.0 has been released:

$ npm install pm2@latest -g
$ pm2 update

Be aware of breaking changes, especially drop support for node 0.12.
Full list here : https://github.com/Unitech/pm2/blob/master/CHANGELOG.md

This is an important issue we really want to solve.
Any feedback and information are welcome.

Feel free to reopen if version 3.0.0 doesn't solve it.

entrptaher · 2018-12-30T13:04:35Z

This caused problem on one of our production server, dump file gets emptied. Exact same problem described as the first post. I should probably re create a new issue, right?

@wallet77

marko-mlinarevic · 2019-03-31T01:02:45Z

were you able to solve it

entrptaher · 2019-03-31T11:13:16Z

@marko-mlinarevic unfortunately I do not think so. But sometimes when we did pm2 update, it would get stuck and no process would generate. After that the pm2 list would be empty. We had to do pm2 resurrect and sometimes it would stay empty, sometimes it would work.

Really curious about such behavior.

nooballday · 2019-04-18T04:29:15Z

I still had this issue, is this behaviour has anything to do with how the Linux system work? i am using Ubuntu 16.04 after reboot the dump file is empty.

tomchomiak · 2019-12-12T04:58:14Z

Running PM2 version 3.5.1 and this issues is still occurring on Ubuntu. I suggest reopening this issue.

Steps to reproduce. Start App and save process list.

pm2 start someAppName
pm2 save

Then show running processes

pm2 status

Then kill PM2 to simulate some type of crash

sudo kill -9 $(pgrep PM2)

Dump file is now gone and trying to resurrect or start the app does not work.

This fails

pm2 start someAppName

As does this

pm2 resurrect

What is going on? Is there a fix in sight for us unlucky souls?

Pashted · 2020-01-24T21:59:37Z

current issue on pm2@4.2.3 is still here
Ubuntu 18.04 LTS (GNU/Linux 4.15.0-20-generic x86_64)

2020-01-25T00:50:40: PM2 log: App [mongodb:1] starting in -fork mode-
2020-01-25T00:50:40: PM2 log: App [mongodb:1] online
2020-01-25T00:50:40: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:40: PM2 log: App [parser:2] starting in -cluster mode-
2020-01-25T00:50:40: PM2 log: App [parser_dev:3] starting in -fork mode-
2020-01-25T00:50:40: PM2 log: App [parser_dev:3] online
2020-01-25T00:50:41: PM2 log: App [parser:2] online
2020-01-25T00:50:41: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:41: PM2 log: App [parser:4] starting in -cluster mode-
2020-01-25T00:50:41: PM2 log: App [parser:4] online
2020-01-25T00:50:41: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:41: PM2 log: App [parser:5] starting in -cluster mode-
2020-01-25T00:50:41: PM2 log: App [parser:5] online
2020-01-25T00:50:41: PM2 log: [Watch] Start watching parser
2020-01-25T00:50:41: PM2 log: App [parser:6] starting in -cluster mode-
2020-01-25T00:50:41: PM2 log: App [parser:6] online
2020-01-25T00:51:13: PM2 log: Stopping app:pm2-logrotate id:0
2020-01-25T00:51:13: PM2 log: Stopping app:mongodb id:1
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:2
2020-01-25T00:51:13: PM2 log: Stopping app:parser_dev id:3
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:4
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:5
2020-01-25T00:51:13: PM2 log: [Watch] Stop watching parser
2020-01-25T00:51:13: PM2 log: Stopping app:parser id:6
2020-01-25T00:51:14: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App name:parser id:6 disconnected
2020-01-25T00:51:14: PM2 log: App [parser_dev:3] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:4] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:6] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:2] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App [parser:5] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: App name:parser id:4 disconnected
2020-01-25T00:51:14: PM2 log: App name:parser id:5 disconnected
2020-01-25T00:51:14: PM2 log: App name:parser id:2 disconnected
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=415 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=724 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=717 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=710 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=703 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=735 msg=process killed
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: pid=702 msg=failed to kill - retrying in 100ms
2020-01-25T00:51:14: PM2 log: App [mongodb:1] exited with code [0] via signal [SIGINT]
2020-01-25T00:51:14: PM2 log: pid=702 msg=process killed
2020-01-25T00:51:14: PM2 log: PM2 successfully stopped
2020-01-25T00:51:39: PM2 log: ===============================================================================
2020-01-25T00:51:39: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2020-01-25T00:51:39: PM2 log: Time                 : Sat Jan 25 2020 00:51:39 GMT+0300 (Moscow Standard Time)
2020-01-25T00:51:39: PM2 log: PM2 version          : 4.2.3
2020-01-25T00:51:39: PM2 log: Node.js version      : 13.3.0
2020-01-25T00:51:39: PM2 log: Current arch         : x64
2020-01-25T00:51:39: PM2 log: PM2 home             : /root/.pm2
2020-01-25T00:51:39: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2020-01-25T00:51:39: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2020-01-25T00:51:39: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2020-01-25T00:51:39: PM2 log: Application log path : /root/.pm2/logs
2020-01-25T00:51:39: PM2 log: Worker Interval      : 30000
2020-01-25T00:51:39: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2020-01-25T00:51:39: PM2 log: Concurrent actions   : 2
2020-01-25T00:51:39: PM2 log: SIGTERM timeout      : 1600
2020-01-25T00:51:39: PM2 log: ===============================================================================
2020-01-25T00:51:41: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2020-01-25T00:51:41: PM2 log: App [pm2-logrotate:0] online

content of /etc/systemd/system/pm2-root.service

[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/root/.nvm/versions/node/v13.3.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid
Restart=on-failure

ExecStart=/root/.nvm/versions/node/v13.3.0/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/root/.nvm/versions/node/v13.3.0/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/root/.nvm/versions/node/v13.3.0/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

batrudinych · 2020-03-27T13:13:10Z

Up the issue. Empty dump file after pm2 (4.2.0) process shutdown.

Pashted · 2020-07-12T13:06:21Z

in 4.4.0 version my problem was solved. thx

jackpittenger · 2020-07-16T04:36:11Z

Had this issue on 4.4.0. Solve it by removing pm2-logrotate, which was deleting the file for no reason

denisenkoaj · 2020-07-17T14:52:57Z

seems that it won't be fixed :D чт, 16 июл. 2020 г. в 10:36, Jack <notifications@github.com>:

…

Having this issue in 4.4.0 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2827 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEJDCQ2ZPLGFNOVT3ZKV54DR3Z7UTANCNFSM4DHFEMCA> .

entrptaher · 2020-07-17T15:49:56Z

It appeared on one of our servers and the dump.pm2 became empty. Maybe this won't be fixed maybe it's very random problem and hard to debug.

jogjech · 2020-08-01T06:57:20Z

Had this issue on 4.4.0. Solve it by removing pm2-logrotate, which was deleting the file for no reason

Same here. The issue was gone after removing pm2-logrotate.

PriyeshGaikwad · 2022-12-13T16:44:46Z

Faced the same issue with version 4.4.0. Below is the fix after started using the logrotate plugin.

Make Change disable autodump like "autodump": "false" in .pm2/module_conf.json

{
"pm2": {
"autodump": "false"
},
"pm2-logrotate": {
"max_size": "2GB",
"retain": "30",
"compress": "true",
"dateFormat": "YYYY-MM-DD_HH-mm-ss",
"workerInterval": "14400",
"rotateInterval": "0 0 * * *",
"rotateModule": true
},
"module-db-v2": {
"pm2-logrotate": {}
}
}

extensionsapp mentioned this issue Apr 20, 2017

PM2 global error caught #2837

Closed

vmarchaud added the S: Need More Data label Apr 20, 2017

extensionsapp changed the title ~~If PM2 is killed, the save file is deleted~~ If PM2 is killed, the dump.pm2 file is empty May 1, 2017

bisubus mentioned this issue Jun 28, 2017

Dump is purged on ENOSPC error #2971

Closed

bisubus mentioned this issue Jul 5, 2017

Dump backup #3001

Merged

denisenkoaj mentioned this issue Feb 21, 2018

pm2 dump loses all entries after unexpected reloading #3485

Closed

Unitech added the S: Pending Release label Mar 2, 2018

wallet77 closed this as completed Jul 4, 2018

If PM2 is killed, the dump.pm2 file is empty #2827

If PM2 is killed, the dump.pm2 file is empty #2827

Comments

extensionsapp commented Apr 10, 2017

vmarchaud commented Apr 20, 2017

extensionsapp commented Apr 20, 2017

Unitech commented Apr 21, 2017

extensionsapp commented Apr 26, 2017

extensionsapp commented May 1, 2017

Unitech commented May 1, 2017

extensionsapp commented May 1, 2017

extensionsapp commented Jun 12, 2017

extensionsapp commented Jun 12, 2017

extensionsapp commented Jul 30, 2017

extensionsapp commented Sep 24, 2017 • edited

extensionsapp commented Sep 26, 2017 • edited

radkodinev commented Oct 22, 2017 • edited

denisenkoaj commented Feb 21, 2018

denisenkoaj commented Feb 21, 2018 • edited

Unitech commented Mar 2, 2018

Unitech commented Apr 9, 2018

shixukai commented Apr 20, 2018

jazarja commented May 18, 2018

dreaganluna commented Jul 2, 2018

radkodinev commented Jul 2, 2018

denisenkoaj commented Jul 3, 2018

wallet77 commented Jul 4, 2018 • edited

entrptaher commented Dec 30, 2018

marko-mlinarevic commented Mar 31, 2019

entrptaher commented Mar 31, 2019

nooballday commented Apr 18, 2019

tomchomiak commented Dec 12, 2019

Pashted commented Jan 24, 2020 • edited

batrudinych commented Mar 27, 2020

Pashted commented Jul 12, 2020

jackpittenger commented Jul 16, 2020 • edited

denisenkoaj commented Jul 17, 2020 via email

entrptaher commented Jul 17, 2020

jogjech commented Aug 1, 2020

PriyeshGaikwad commented Dec 13, 2022

extensionsapp commented Sep 24, 2017 •

edited

extensionsapp commented Sep 26, 2017 •

edited

radkodinev commented Oct 22, 2017 •

edited

denisenkoaj commented Feb 21, 2018 •

edited

wallet77 commented Jul 4, 2018 •

edited

Pashted commented Jan 24, 2020 •

edited

jackpittenger commented Jul 16, 2020 •

edited