‌Usage with mqemiiter-redis

[behrad]

I want to share a redis mqemitter in scaled-up aedes processes @mcollina
Can you guide me how to use a redis powered mqemitter shared with child processes?

[mcollina]

just use mqemitter-redis everywhere. This is more if you want a bunch of child_processes to communicate.

[behrad]

our subscription tree is so huge, when mqemitter-redis loads the tree in qlobber (3.5G RAM) we can't afford a 3.5*cores GIG memory machine :(

[mcollina]

I understand: I would recommend you run a mqemitter-redis in the process master, and then you can use the 'cluster' module for core with your aedes instances to share the same port. The child processes are allowed to fail and restart.

If you look at the code, you can run an instance-level daemon with https://github.com/mcollina/mqemitter-cs, and exposes it on a local socket. So, your daemon will run your mqemitter-redis topic cache, and it will stays pretty stable and you can redeploy your aedes instances in the background.

I'm happy to jump on a quick call to discuss.

It might well be that you need something similar to the design here with the persistence layer, as its the persistence that holds all the subscription data: https://github.com/mcollina/aedes-cached-persistence/blob/master/index.js#L27.

(we might even think about having those persisted in a level instance)

[behrad]

I'm happy to jump on a quick call to discuss

it will be great 👍

Let me dig into your comment first!

It might well be that you need something similar to the design here with the persistence layer, as its the persistence that holds all the subscription data: https://github.com/mcollina/aedes-cached-persistence/blob/master/index.js#L27.

Uh! My mistake!!! mqemitter-redis has no topic in-memory cache, This is the redis-persistence that is our bottleneck!
How about trying to implement a process-shared-aedes-redis-persistence on top of current redis-persistence with your suggestion above?

[mcollina]

@behrad it should be doable. Have a look at how https://github.com/mcollina/mqemitter-cs/blob/master/index.js works. The key is to use https://github.com/mcollina/tentacoli, which provides a request/response + streams multiplexing transport.

[behrad]

(we might even think about having those persisted in a level instance)

What about the idea of caching the hole subscriptions inside filesystem (instead of memory).
Something like a very fast file system based Qlobber using nedb/level? I don't think this as a performant solution however.

[behrad]

@behrad it should be doable. Have a look at how https://github.com/mcollina/mqemitter-cs/blob/master/index.js works. The key is to use https://github.com/mcollina/tentacoli, which provides a request/response + streams multiplexing transport.

You're talking about extending aedes-cached-persistence to an aedes-cs-persistence to be used as a shared process cached Qlobber to talk to the higher persistence implementation @mcollina ?

[mcollina]

I am talking about writing a new module that implements aedes-persistence that does that.

[behrad]

I played with tentacoli and a 100 req/rep batch took 8secs. I also ran the bench.js inside the project and it took 13.5secs.
It doesn't feel performant for a cs-persistence implementation where we need fast Qlobber matching. (i.e. cs-persistence talks to a redis-persistence[with cached-persistence inside] )

Am i missing something @mcollina ?

We need a multi thousand per sec req/rep multiplexing. What about nanomsg-node?

[mcollina]

@behrad there is currently a performance regression on readable-stream 2.3, which tentacoli is using (nodejs/readable-stream#304). The number I have is around 23.5 req/ms, which we can approximate to 20000, which is close to the performance of pure HTTP in Node.

In order to be faster you will need to use https://github.com/mafintosh/protocol-buffers

[behrad]

@behrad there is currently a performance regression on readable-stream 2.3, which tentacoli is using (nodejs/readable-stream#304). The number I have is around 23.5 req/ms, which we can approximate to 20000

Are you telling that nodejs/readable-stream#304 will improve x2000 from 0.007 req/ms to 23 req/ms !?

which is close to the performance of pure HTTP in Node

Why not go ipc instead? like node-ipc or even pure child_process ipc ? We can have a very simple req/rep on top of that @mcollina

In order to be faster you will need to use https://github.com/mafintosh/protocol-buffers

Considering using native JSON (instead of msgpack) I don't think we need that, huh?

[mcollina]

Are you telling that nodejs/readable-stream#304 will improve x2000 from 0.007 req/ms to 23 req/ms !?

On my box I have (with readable-stream@2.2):

benchPingPong*100: 4249.158ms

Each of those runs sends 1000 messages back and forth: 100 * 1000 req / 4249 ms = 23.534 req/ms.

Why not go ipc instead? like node-ipc or even pure child_process ipc ? We can have a very simple req/rep on top of that @mcollina

You need streaming capability. req/res would not be enough, or it is a lot of work to support the API we currently have for that.

[behrad]

You need streaming capability

Any reason other than back pressure support?
Do we need more than the below API?

1. persistence client instance --addSubToQlobber--> persistence master (without reply)
2. persistence client instance --matchTopics--> persistence master (with reply, here we need streaming the whole matched topics?)
3. 1. persistence client instance --removeSubFromQlobber--> persistence master (without reply)

[mcollina]

I mean something different. I think we should expose the entire aedes-persistence API over the wire, so it can be used with any real implementation.

[behrad]

I think we should expose the entire aedes-persistence API over the wire

How that scales?
Main rationale behind that was sharing the Qlobber to reduce memory when scaling up, however store/delete subscriptions can be done concurrently by each process. Collecting them all inside the master, worries me. Master will melt. considering the expensive, cpu-intesive Qlobber matching that it should handle at the same time.

so it can be used with any real implementation

I trade off design/modularity for performance here, since I've tested underlying mosca/aedes persistence performance/issues over a year.

[behrad]

20000, which is close to the performance of pure HTTP in Node

Have you any example of a node HTTP server code which handles 20k/sec single process?

[mcollina]

Check out https://github.com/fastify/fastify.

For what you want to achieve, you can get something better with protocol-buffers. It would be good to have this as a an option in aedes-cached-persistence.

[behrad]

however store/delete subscriptions can be done concurrently by each process. Collecting them all inside the master, worries me. Master will melt. considering the expensive, cpu-intesive Qlobber matching that it should handle at the same time.

I am not getting the protocol-buffers idea in this regard, as that is a format system !?

[mcollina]

protocol-buffer is a data format. Most of the processing time related to distributed system is due to parsing and generating the messages to send. Il giorno mer 28 giu 2017 alle 20:09 Behrad <notifications@github.com> ha scritto:

…

however store/delete subscriptions can be done concurrently by each process. Collecting them all inside the master, worries me. Master will melt. considering the expensive, cpu-intesive Qlobber matching that it should handle at the same time. I am not getting the protocol-buffers idea in this regard, as that is a format system !? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADL456HTN4S6HCgdfzSJL2HimygKZIeks5sIpbAgaJpZM4Nlfd0> .

[behrad]

Most of the processing time related to
distributed system is due to parsing and generating the messages to send

Yea! but we can also switch to any format, test them, after distribution interface clarified. My main concern is to only have rpc/ipc for Qlobber and let the whole persistence api work in process (local)

[mcollina]

👍