pm2 not resurrecting after reboot - Linux Mint, pm2 3.2.2

[jcollum]

Same machine as this issue: #3635, new version of pm2 has same issue.

PM2 is not restarting a saved process after a reboot.

$ pm2 --version
3.2.2

$ pm2 start /home/justin/mining/eth.sh
[PM2] Starting /home/justin/mining/eth.sh in fork_mode (1 instance)
[PM2] Done.
┌──────────┬────┬─────────┬──────┬──────┬────────┬─────────┬────────┬─────┬──────────┬────────┬──────────┐
│ App name │ id │ version │ mode │ pid  │ status │ restart │ uptime │ cpu │ mem      │ user   │ watching │
├──────────┼────┼─────────┼──────┼──────┼────────┼─────────┼────────┼─────┼──────────┼────────┼──────────┤
│ eth      │ 0  │ N/A     │ fork │ 9748 │ online │ 0       │ 0s     │ 0%  │ 3.0 MB   │ justin │ disabled │
└──────────┴────┴─────────┴──────┴──────┴────────┴─────────┴────────┴─────┴──────────┴────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app


$ pm2 startup
[PM2] Init System found: systemd
[PM2] To setup the Startup Script, copy/paste the following command:
sudo env PATH=$PATH:/home/justin/.nvm/versions/node/v10.13.0/bin /home/justin/.nvm/versions/node/v10.13.0/lib/node_modules/pm2/bin/pm2 startup systemd -u justin --hp /home/justin


$ sudo env PATH=$PATH:/home/justin/.nvm/versions/node/v10.13.0/bin /home/justin/.nvm/versions/node/v10.13.0/lib/node_modules/pm2/bin/pm2 startup systemd -u justin --hp /home/justin
[sudo] password for justin:
Sorry, try again.
[sudo] password for justin:
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=justin
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/home/justin/.nvm/versions/node/v10.13.0/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/home/justin/.pm2
PIDFile=/home/justin/.pm2/pm2.pid

ExecStart=/home/justin/.nvm/versions/node/v10.13.0/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/home/justin/.nvm/versions/node/v10.13.0/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/home/justin/.nvm/versions/node/v10.13.0/lib/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Target path
/etc/systemd/system/pm2-justin.service
Command list
[ 'systemctl enable pm2-justin' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-justin.service
[PM2] Making script booting at startup...
[PM2] [-] Executing: systemctl enable pm2-justin...
[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
$ pm2 save

[PM2] Remove init script via:
$ pm2 unstartup systemd


$ pm2 save
[PM2] Saving current process list...
[PM2] Successfully saved in /opt/.pm2/dump.pm2

$ sudo reboot

#
#
# MACHINE REBOOTS
#
#

$ pm2 ls
[PM2] Spawning PM2 daemon with pm2_home=/opt/.pm2
[PM2] PM2 Successfully daemonized
┌──────────┬────┬─────────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴─────────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app


$ pm2 report
===============================================================================
--- PM2 REPORT (Sat Nov 17 2018 14:13:07 GMT-0800 (Pacific Standard Time)) ----
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.2.2
node version         : 10.13.0
node path            : /home/justin/.nvm/versions/node/v10.13.0/bin/pm2
argv                 : /home/justin/.nvm/versions/node/v10.13.0/bin/node,/home/justin/.nvm/versions/node/v10.13.0/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : justin
uid                  : 1000
gid                  : 1000
uptime               : 3min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.2.2
node version         : 10.13.0
node path            : /home/justin/.nvm/versions/node/v10.13.0/bin/pm2
argv                 : /home/justin/.nvm/versions/node/v10.13.0/bin/node,/home/justin/.nvm/versions/node/v10.13.0/bin/pm2,report
argv0                : node
user                 : justin
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Core(TM)2 Duo CPU     E8400  @ 3.00GHz
cpus nb              : 2
freemem              : 7672913920
totalmem             : 8370278400
home                 : /home/justin
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────┬────┬─────────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴─────────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
┌──────────┬────┬─────────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴─────────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

$ node --version
v10.13.0

[jcollum]

@wallet77 you helped on the previous issue, maybe this one will have the information you need to solve it

[KaiSchwarz-cnic]

adding myself here as watcher too. we are using pm2 v3.2.3 on a freshly installed ubuntu 18.
we use NVM (0.33.11) to have nodejs (v10.14.2) running under a separate user "node". we generated the startup script, started our apps successfully and have created the dump file. we rebooted - but the daemon did not came up.
after having the startup script again generated (after having it uninstalled), we created again a dump -> it worked.

What I also recognized: I generated a dump file using pm2 save that was NOT minified. after having that startup script regenerated, we created a new dump. this one was minified. No idea what is behind this issue. We could reproduce this behavior on two servers.

NOTE: the initial installation of pm2 used v3.2.2 and later on we auto-upgraded to v3.2.3. so the first startup script might have been created using v3.2.2 and the current one with v3.2.3. No idea if something has changed between these to versions that affects the startup script generator or creating dump files. At least I have not found something in the release overview.

[jcollum]

Thanks for adding that @Papakai , hopefully it will help

[KaiSchwarz-cnic]

Guys, any progress in this matter? This is affecting our production servers still in newest release.
Our plan is to downgrade to have this feature back again - I really dislike this step, but just makes sense as it doesn't look like you get this fixed as this issue is open since 17 Nov 2018.

I for sure know that this is a opensource project and that we can't expect things to get fixed in days, but months?

[KaiSchwarz-cnic]

Hey again,

sorry from my side, I got the chance to have an eye on our server setup playbooks. --hp was missing there as parameter for the startup script generator.
Result: pm2 tried to lookup the dump under root home directory, which is for sure wrong and was the issue here.
Sorry for having pushed this issue, I expected the ansible playbook to have been realized as I requested internally. maybe this helps others.

[jcollum]

--hp was missing there as parameter for the startup script generator.

What did you set it to? Did that fix the issue? Maybe this is just a documentation problem...

[KaiSchwarz-cnic]

for security reasons a nodejs app should not run under root user (if the process is getting hacked, the hacker has administrative rights). therefore using a specific user for nodejs apps ftw.!
The pm2 startup script generator has to be run as root by providing this user in command line argument -uand its home directory path in command line argument --hp.
The command line arguments are documented here, but yes it could be useful to tell people why they should not run node apps using root user - even though this should be basic knowledge.

Let's say you have the os usre "node" for that purpose, you could have something like this:
root@...> pm2 startup ubuntu -u node --hp /home/node
Later (and because of this) you will find the pm2 folder under /home/node/.pm2.

In your issue description you might recognize that after reboot your pm2 ls outputs a different home directory then the one you've set previously. That's why the dump file can't be found (produced by pm2 save) and thus no apps can be restarted after reboot.
No exact idea, instead of sudo try to execute the command without sudo directly under root user. At least for us this worked in the past. I will link in another github issue where I described the process a bit, not sure if 100% correctly as I was in hurry.

[KaiSchwarz-cnic]

Here it is.

[jcollum]

for security reasons a nodejs app should not run under root user (if the process is getting hacked, the hacker has administrative rights). therefore using a specific user for nodejs apps ftw.!

@Papakai do you have a guide that you like for creating this user? I've seen many suggestions for how to do it, enough that I'm not sure which is the correct one for this problem

edit: I've been messing with this for about an hour now and running into enough problems that I'm going to give up and wait for a response. I need to be able to run node 10, pm2 and a bash script that will restart on startup. That's all I'm trying to achieve.

For instance, when I run $ sudo env PATH=$PATH:/home/nodejs/.nvm/versions/node/v10.15.1/bin ...etc I get a password challenge, but I created the user without a password. Ideally I want a user that can run pm2 scripts that will be restarted when the sytem restarts but is not able to login via ssh. I think that's the safest way to run scripts with pm2 on a remote machine.

[KaiSchwarz-cnic]

@jcollum sorry, but I am not a server-/workstation administrator, we always get our machines freshly installed and ready to use. Therefore I have not instructions on how to create such an user.
But I think sudo useradd -d /home/node -m node does the trick. This user won't have administrative rights.

I suggest to install and use NVM under that user. We use nvm also in production. If you're familiar with nvm, you will recognize that especially in development and staging area, it is very useful to switch nodejs versions without overhead just to be able to test new versions.

There are several things that you need to be aware of:
a) A non-root user cannot open Port 80 or Port 443 for example, just because of the missing rights.
-> Start your nodejs app on Port 8080 or 8443 and use a Load Balancer in front to map the correct port to the one of your app. Another possibility is using ip tables. Maybe there are further possibilities... At least we used both in the past and they worked like a charm. But don't ask me technical details around this, because I am not a system administrator.
b) Using pm2 based on NVM: the startup script is getting generated with a nodejs-version related path and every time when you upgrade nodejs, you will have to

• re-trigger installation of pm2 startup script (maybe creating a new pm2 dump could also be the case)
• remember to reinstall all globally installed nodejs packages (there's also a command line argument for nvm to care about this automatically)

That's all I am aware of.

Concerning your installation issues... I can't provide you more than the above git issue. That was my past, in the meanwhile things have changed in our company so that I am getting machines that are completely ready to use with the mentioned setup. I can just say that it works fine, just if you know and care about the above points.

/EDIT:
Just that you also know about this possibility (which works, but honestly I prefer keeping everything as explained above):
https://thomashunter.name/posts/2012-12-04-drop-root-privileges-in-node-js

  srv.listen(24, function() {
    console.log('listening');
    process.setgid('www-data');
    process.setuid('www-data');
  });

This downgrades user rights by switching to the given group id and userid. As you may recognize, the author of the provided article also points out that this might not follow industry best practices.
At least with nvm / pm2 / nodejs you have a valid setup.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.