Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"pm2 reload" while reload in progress leads to zombie processes and pm2 errors #2951

Closed
migg24 opened this issue Jun 19, 2017 · 4 comments
Closed

Comments

@migg24
Copy link

migg24 commented Jun 19, 2017

What's going wrong?
When you send pm2 a pm2 reload app while a pm2 reload app is still in progress it leads to errors and zombie processes of the app that are not visible in pm2 list but in ps aux | grep node. The pm2 daemon IMHO should not reload the app if a reload is in progress.

How could we reproduce this issue?

  1. Start an app with 8 instances, give it a kill timeout of 10000 and a 5000 timeout on SIGINT to exit the process.
  2. Stream pm2 logs
  3. Open 2 more terminals on the same machine
  4. Execute pm2 reload app on terminal 2, and shortly after that execute pm2 reload app on terminal 3

In my test it resulted in this error:

[PM2][ERROR] Process 6 not found

And the pm2 reload stopped, but this error starts to fill the pm2.log:

Unknown id _old_6

After that there are 8 instances in pm2 list but more than 8 in ps aux | grep node and they seem to still respond to connections so that some are working on an old version of the code and some on the new version if there is a code change that triggered the reload.

Supporting information

PM2 version: 2.5.0, 2.1.6
Node version: 4.8.3
Linux

I think the only option to handle this is to no let the pm2 god daemon reload if it is already reloading. But maybe there are other ideas/options.

@migg24 migg24 changed the title pm2 reload while reload in progress leads to zombie processes and pm2 errors "pm2 reload" while reload in progress leads to zombie processes and pm2 errors Jun 19, 2017
@Unitech Unitech added the T: Bug label Jun 26, 2017
Unitech added a commit that referenced this issue Jun 27, 2017
@Unitech
Copy link
Owner

Unitech commented Jun 27, 2017

pm2 reload lock via timestamped lockfile added, to try it:

$ npm install Unitech/pm2#development -g
$ pm2 update
# Try to start two pm2 reload in same time  
$ pm2 reload all & pm2 reload all

@kschingiz
Copy link

@Unitech It didn't helped me, it created another instance of app, so in pm2 status it shows 2 instances, but node processes 4, was 3 before you instructions.
I have 2 instances with latest update of app, but 2 is deprecated, and my clients redirecting to older instances some time, it creates me bugs and crashes of mobile apps.
How can I solve it?

Unitech added a commit that referenced this issue Jul 4, 2017
@Unitech
Copy link
Owner

Unitech commented Jul 20, 2017

Published on PM2 2.6.0:

$ npm install pm2@latest -g
$ pm2 update

@Unitech Unitech closed this as completed Jul 20, 2017
@migg24
Copy link
Author

migg24 commented Aug 11, 2017

Big thanks for this quick fix! We tested it in our environment and the problem does not appear anymore.
👍

inerc pushed a commit to inerc/pm2 that referenced this issue Feb 11, 2020
inerc pushed a commit to inerc/pm2 that referenced this issue Feb 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants