restarted instances on non-primary ports don't listen

[BenLubar]

NodeBB version 1.1.0, git hash 8584f9a

1. NodeBB occasionally hangs for us. We haven't found the culprit yet, but we have a script that kills an instance if it doesn't respond to a request within 5 seconds.
2. Our build script changes the code that sets the cache buster to a random UUID to use a random hex string assigned during upgrades.
3. When the primary instance (4567) restarts, users receive a "the forum has been updated, click here to refresh" notification.
4. When the secondary instance (4568) restarts, it waits until the primary instance gives the "all clear" signal, which won't happen if the primary instance has already given the signal.

Expected: the loader.js process remembers that the preparation of JS and CSS has already been run and immediately tells the secondary instance it can continue.
Actual: the secondary app.js process waits indefinitely until the primary app.js process restarts and runs the preparation.

Expected: restarted instances don't trigger the update notification if the cache buster is the same as it was previously.
Actual: restarting an instance always triggers the update notification, regardless of what the cache buster is.

[boomzillawtf]

And by "hangs for us," means that the node process sits at 100% CPU usage according to top.

[julianlam]

Is this "the reason" your NodeBB goes down?

[BenLubar]

No, but it's the reason it makes noise when it comes back up.

[boomzillawtf]

Well, sort of. I mean...it gets stuck with 100% CPU doing something. There's no pattern to where it seems to get stuck. There are some gzipped tarballs here with linux perf tools stack traces and flamegraphs:

https://what.thedailywtf.com/topic/19990/the-official-server-cooties-tracking-thread

Those are a representative sampling of what we've seen when this happens. The process that's "stuck" somewhere is busy doing whatever it's doing and not responding to other requests. So as far as the user is concerned, the site is down (or half down, if you're talking to the unlucky process and it wasn't both processes that were pegging).

We just have no idea why it would be getting stuck.

[barisusakli]

What nodejs version and does it still happen when you disable all custom functionality?

[xidui]

@BenLubar
Waiting forever?
I had the experience that I will wait for several minutes with no response but 100% CPU of nodejs.
It recovers after about 3 minutes.

Not only restarts nodebb, but also restart nginx causes this as well.

My nodejs version is 4.4.3

[BenLubar]

@xidui in this case, it was waiting for a templates:compiled message that was never sent. This isn't the bug that causes 100% CPU usage and unresponsiveness - this is before the instance starts listening for connections.