Zero Downtime deployment

[rajatdhoot123]

Summary

I’ve deployed a Next.js application using PM2 in cluster mode with max instances. The setup is part of a distributed system where the app runs on multiple servers, all behind an NGINX reverse proxy.

My assumption was that reloading PM2 (e.g., using pm2 reload) would gracefully restart workers one by one, ensuring zero downtime. However, I'm still encountering 502 errors during deployment.

Any suggestions on how to achieve zero-downtime deployments with PM2 in this setup?

MY PM2 setup below

module.exports = {
  apps: [
    // ─── quickmake blue/green ────────────────────────────────
    {
      name: "project",
      script: "./node_modules/next/dist/bin/next",
      args: "start",
      exec_mode: "cluster",
      instances: "max",
      listen_timeout: 120000,
      env_production: {
        PORT: 3000,
        NODE_ENV: "production",
      },
    }
   ]
  },

[icyJoseph]

It's been a while now since I worked with pm2... but if you are in cluster mode, these share the same .next output folder, right?

I'd guess, that when you swap it for a new .next, some of the workers are still "trying" to use the old version, and error out?

Maybe this old thread can point you in the right direction: #10675 (comment)

[rajatdhoot123]

If you don't make the build and do pm2 reload all still get 502 for sometime.

[icyJoseph]

Are there any logs we could use here? What's our working assumption for this error, that on reload a connection arrives to a cluster worker that just got reloaded? Shouldn't pm2 route it to an already finished, or not started worker? Is the worker getting shutdown as it serves a response?

[rajatdhoot123]

Logs added

2025-06-16T07:48:02: PM2 log: App [quickmake:0] starting in -cluster mode-
2025-06-16T07:48:02: PM2 log: App [quickmake:2] starting in -cluster mode-
2025-06-16T07:48:03: PM2 log: App [quickmake:0] online
2025-06-16T07:48:03: PM2 log: App [quickmake:2] online
2025-06-16T07:48:03: PM2 log: -reload- New worker listening
2025-06-16T07:48:03: PM2 log: Stopping app:quickmake id:_old_0
2025-06-16T07:48:03: PM2 log: -reload- New worker listening
2025-06-16T07:48:03: PM2 log: Stopping app:quickmake id:_old_2
2025-06-16T07:48:03: PM2 error: AXM MONITOR Unknown id _old_221
2025-06-16T07:48:03: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:03: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:03: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:03: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:03: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:03: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 error: AXM MONITOR Unknown id _old_219
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:04: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:05: PM2 error: AXM MONITOR Unknown id _old_221
2025-06-16T07:48:05: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:05: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:05: PM2 log: pid=3130312 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:05: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:05: PM2 log: App name:quickmake id:_old_0 disconnected
2025-06-16T07:48:05: PM2 log: App [quickmake:_old_0] exited with code [0] via signal [SIGINT]
2025-06-16T07:48:05: PM2 log: Process with pid 3130312 still alive after 1600ms, sending it SIGKILL now...
2025-06-16T07:48:05: PM2 log: pid=3130326 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:05: PM2 log: Process with pid 3130326 still alive after 1600ms, sending it SIGKILL now...
2025-06-16T07:48:05: PM2 error: AXM MONITOR Unknown id _old_219
2025-06-16T07:48:05: PM2 log: App name:quickmake id:_old_2 disconnected
2025-06-16T07:48:05: PM2 log: App [quickmake:_old_2] exited with code [0] via signal [SIGKILL]
2025-06-16T07:48:05: PM2 log: pid=3130312 msg=process killed
2025-06-16T07:48:05: PM2 log: pid=3130326 msg=process killed
2025-06-16T07:48:05: PM2 log: App [quickmake:3] starting in -cluster mode-
2025-06-16T07:48:05: PM2 log: App [quickmake:4] starting in -cluster mode-
2025-06-16T07:48:05: PM2 log: App [quickmake:3] online
2025-06-16T07:48:05: PM2 log: App [quickmake:4] online
2025-06-16T07:48:06: PM2 error: AXM MONITOR Unknown id _old_221
2025-06-16T07:48:06: PM2 log: -reload- New worker listening
2025-06-16T07:48:06: PM2 log: Stopping app:quickmake id:_old_4
2025-06-16T07:48:06: PM2 log: -reload- New worker listening
2025-06-16T07:48:06: PM2 log: Stopping app:quickmake id:_old_3
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 error: AXM MONITOR Unknown id _old_219
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:06: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 error: AXM MONITOR Unknown id _old_221
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 error: AXM MONITOR Unknown id _old_219
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=failed to kill - retrying in 100ms
2025-06-16T07:48:07: PM2 log: Process with pid 3130348 still alive after 1600ms, sending it SIGKILL now...
2025-06-16T07:48:07: PM2 log: Process with pid 3130337 still alive after 1600ms, sending it SIGKILL now...
2025-06-16T07:48:07: PM2 log: App name:quickmake id:_old_3 disconnected
2025-06-16T07:48:07: PM2 log: App [quickmake:_old_3] exited with code [0] via signal [SIGKILL]
2025-06-16T07:48:07: PM2 log: App name:quickmake id:_old_4 disconnected
2025-06-16T07:48:07: PM2 log: App [quickmake:_old_4] exited with code [0] via signal [SIGKILL]
2025-06-16T07:48:07: PM2 log: pid=3130348 msg=process killed
2025-06-16T07:48:07: PM2 log: pid=3130337 msg=process killed

[rajatdhoot123]

You can ignore this error this 2025-06-16T07:48:07: PM2 error: AXM MONITOR Unknown id _old_219 is from some corrupt data adding fresh logs also

2025-06-16T07:53:27: PM2 log: Process 2 in a stopped status, starting it
2025-06-16T07:53:27: PM2 log: App [quickmake:2] starting in -cluster mode-
2025-06-16T07:53:27: PM2 log: App [quickmake:0] online
2025-06-16T07:53:27: PM2 log: App [quickmake:2] online
2025-06-16T07:53:27: PM2 log: Process 3 in a stopped status, starting it
2025-06-16T07:53:27: PM2 log: App [quickmake:3] starting in -cluster mode-
2025-06-16T07:53:27: PM2 log: Process 4 in a stopped status, starting it
2025-06-16T07:53:27: PM2 log: App [quickmake:4] starting in -cluster mode-
2025-06-16T07:53:27: PM2 log: App [quickmake:3] online
2025-06-16T07:53:27: PM2 log: App [quickmake:4] online
2025-06-16T07:53:47: PM2 log: App [quickmake:0] starting in -cluster mode-
2025-06-16T07:53:47: PM2 log: App [quickmake:2] starting in -cluster mode-
2025-06-16T07:53:47: PM2 log: App [quickmake:0] online
2025-06-16T07:53:47: PM2 log: App [quickmake:2] online
2025-06-16T07:53:48: PM2 log: -reload- New worker listening
2025-06-16T07:53:48: PM2 log: Stopping app:quickmake id:_old_0
2025-06-16T07:53:48: PM2 log: -reload- New worker listening
2025-06-16T07:53:48: PM2 log: Stopping app:quickmake id:_old_2
2025-06-16T07:53:48: PM2 log: App name:quickmake id:_old_2 disconnected
2025-06-16T07:53:48: PM2 log: App [quickmake:_old_2] exited with code [0] via signal [SIGINT]
2025-06-16T07:53:48: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:48: PM2 log: pid=3134676 msg=process killed
2025-06-16T07:53:48: PM2 log: App [quickmake:3] starting in -cluster mode-
2025-06-16T07:53:48: PM2 log: App [quickmake:3] online
2025-06-16T07:53:48: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:48: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:48: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: -reload- New worker listening
2025-06-16T07:53:49: PM2 log: Stopping app:quickmake id:_old_3
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: App name:quickmake id:_old_3 disconnected
2025-06-16T07:53:49: PM2 log: App [quickmake:_old_3] exited with code [0] via signal [SIGINT]
2025-06-16T07:53:49: PM2 log: pid=3134689 msg=process killed
2025-06-16T07:53:49: PM2 log: App [quickmake:4] starting in -cluster mode-
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: App [quickmake:4] online
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:49: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:50: PM2 log: pid=3134675 msg=failed to kill - retrying in 100ms
2025-06-16T07:53:50: PM2 log: -reload- New worker listening
2025-06-16T07:53:50: PM2 log: Stopping app:quickmake id:_old_4
2025-06-16T07:53:50: PM2 log: Process with pid 3134675 still alive after 1600ms, sending it SIGKILL now...
2025-06-16T07:53:50: PM2 log: App name:quickmake id:_old_4 disconnected
2025-06-16T07:53:50: PM2 log: App [quickmake:_old_4] exited with code [0] via signal [SIGINT]
2025-06-16T07:53:50: PM2 log: App name:quickmake id:_old_0 disconnected
2025-06-16T07:53:50: PM2 log: App [quickmake:_old_0] exited with code [0] via signal [SIGKILL]
2025-06-16T07:53:50: PM2 log: pid=3134690 msg=process killed
2025-06-16T07:53:50: PM2 log: pid=3134675 msg=process killed

[rajatdhoot123]

@icyJoseph Let me know if something else is required from my end.