100% CPU and server crashes after connecting 10k clients

[eladnava]

Hi @mcollina =)

Getting a strange crash and CPU halt with the latest Aedes (0.21.0):

/var/app/current/node_modules/aedes/node_modules/reusify/reusify.js:7
  function get () {
               ^

RangeError: Maximum call stack size exceeded
    at Object.get (/var/app/current/node_modules/aedes/node_modules/reusify/reusify.js:7:16)
    at SubscribeState.compiled (/var/app/current/node_modules/aedes/node_modules/fastfall/fall.js:38:25)
    at SubscribeState.doSubscribe (/var/app/current/node_modules/aedes/lib/handlers/subscribe.js:34:16)
    at makeCall (/var/app/current/node_modules/aedes/node_modules/fastseries/series.js:113:10)
    at SubscribeState.ResultsHolder.release (/var/app/current/node_modules/aedes/node_modules/fastseries/series.js:94:9)
    at work (/var/app/current/node_modules/aedes/node_modules/fastfall/fall.js:121:25)
    at sendRetained (/var/app/current/node_modules/aedes/lib/handlers/subscribe.js:107:5)
    at SubscribeState.subTopic (/var/app/current/node_modules/aedes/lib/handlers/subscribe.js:102:5)
    at work (/var/app/current/node_modules/aedes/node_modules/fastfall/fall.js:105:23)
    at /var/app/current/node_modules/aedes/lib/handlers/subscribe.js:64:5

Before the server crashes, the process consumed 100% CPU and is completely unresponsive.

This happens after about 10k clients connect. Keepalive interval is set at 5 minutes. Using MemoryPersistence.

Most clients do not subscribe to anything, by the way.

Oh, and I set the concurrency to 10,000. Didn't help.

[mcollina]

Yes, this is perfectly "normal", and described in #60.
Most of the code is highly synchronous for performance reason, but it might use a process.nextTick in a couple of places (like in the memory persistence).

PR is welcomed.

[eladnava]

@mcollina It's normal that Aedes is unable to handle 10k connections? Am I doing something wrong?

Or is the MemoryPersistence unable to handle 10k connections?

[mcollina]

It's normal that it crashes with RangeError: Maximum call stack size exceeded.
It's not looping, Aedes works heavily on the stack, specially with the in-memory mq and persistence.

node --stack-size=X solves this, where X is big enough.

wrapping the callbacks in process.nextTick in the memory persistence will probably solve your issue.

[eladnava]

I ended up going with mqtt-connection since I don't need any kind of persistence. Seems to work well so far for me.

[mcollina]

Perfect, I'm closing this now.

[GavinDmello]

Hey, I'm facing a similar issue after 15k connections.The process becomes unresponsive for quite some time and then starts working again. I'm not sure if gc is the culprit here as it is synchronous and takes quite a bit of time.

[mcollina]

@GavinDmello with the same error? what is the memory usage? Can you produce a script to reproduce artificially?

[GavinDmello]

The server doesn't crash in my case and memory is well under control. It's just the CPU which is cranking up to 100 % and stays there with nothing happening(no logs) and comes back to normal after some time.

[mcollina]

@GavinDmello if I can reproduce, I can help fixing it. Tools like http://npm.im/0x, dtrace, perf, etc, are built to solve those kind of issues.

[cordovapolymer]

related to #88