Redis, RabbitMQ, Kafka adapters

[nmn]

Channels are an amazingly versatile transport mechanism to pass values between go blocks. In clojure, and go, they are also used to pass values between threads.

Since javascript has no threads, channels can and should be used to pass values between node processes. Since processes don't have any shared memory, this needs to be done using something like Redis to manage the common state.

Building a simple API to build various adapters for channels would make it an invaluable tool for multi-process management in Node.

Additionally, the same API should work for communication between master process and child process. And between the main process and web workers on the client-side.

Please let me know, how I can help with this.

[nmn]

Just wanted to add something here:

The main challenge with the adapters is the way to block on put. The RxJS/Observable way of fire and forget is relatively trivial to implement.

[zeroware]

I like the idea of adapters but I don't know if the name is appropriate.
I think that we need to decide about the error handling (#14) before implementing things.

[jlongster]

I don't fully understand the problem; you just need to implement whatever kind of inter-process communication you want, and provide a channel as the interface. I wouldn't implement the channel interface, I would just provide a real channel that the backend listens for puts and takes and dispatches those to the backend implementation of sending/receiving values.

[zeroware]

@jlongster Yes I agree. This kind of "adapters" has nothing to do with the core lib. We can just have another module that can take i/o from and to channels.

[nmn]

@jlongster @zeroware I think I wasn't clear in that I don't think the adapters themselves should be a part of the core lib.

I've been doing some research on this. Most solutions for inter-process communication, as well as most async solutions for javascript have one thing in common — fire and forget.

RXjs, BaconJS, Redis, Web Workers, Node Child Processes etc. In all cases, when you fire an event or publish a value on a stream, you don't wait for a listener.

Channels are unique in their ability to block on put. I think there should be an official way to map the usual fire and forget method to a block on put.

So far the only solution I can think of is to send a message and then wait for a confirmation response. Would love to hear your thoughts.