Skip to content

Loading…

Redis PubSub vs Multiplexed Firehose #236

Open
timothyjoelwright opened this Issue · 18 comments

7 participants

@timothyjoelwright

Currently SS multiplexes all events over a single redis channel. If an app were to scale large enough, it's easily possible for 100,000's of messages to be sprayed to every SS instance - taking up bandwidth and burdening the local SS instance with determining if each message should be propagated locally or not.

I spent a few hours today creating a version of the Redis publish transport that properly uses Redis' pub/sub to route messages only to the servers that can use them. Hopefully this will allow more flexibility when it's necessary to scale Redis.

I modified two files:

src/publish/transports/redis.coffee
src/utils/unique_set.coffee

Since it's more of a working proof-of-concept rather than something that's pull-able, here's a gist of the files:
https://gist.github.com/2779421

I've done very basic testing and everything works as expected with multiple clients connected to several SS servers and a common Redis server. There may be a little bit of leaking where redis channels associated with socketIDs may not be unsubscribed when the socket disconnects. I didn't see it happening via MONITOR, but if they're ever removed from the unique_set, then the events should fire and the channel for the socketId will be unsubscribed.

@owenb

Hey there

Sorry for the delay getting to this.

This is an interesting idea. I am certainly wanting SocketStream to be used in really large scale projects, so I can see the value in doing this. That said, the potential bandwidth savings must be balanced with the additional code complexity (and performance implications of calling an event emitter each time something changes in UniqueSet).

Rather than trying to replace what's there (at least for now); I think it would be worth putting this code into a separate module which could be included in your app.js with:

ss.publish.transport.use(require('redis-demultiplexed'))

(or whatever you decide to call it). You would still need to patch unqiue_set.js for now, but if you fancy making it into a module I'll happily test it on my projects and try to see if I can identify any problems.

Thanks very much for the idea + code.

Owen

@timothyjoelwright

Thanks for your feedback. To respect your preferences for minimal code complexity and performance impact, I'll remove the event emitter and look for the best places to add generic subscribe/unsubscribe function calls to the transport instead. I plan on using this in production for a non-critical project, so any testing you'll be able to offer with the finished module is appreciated!

@owenb

Hey @timothyjoelwright

I have given this a bit more thought. I am pondering taking out Redis pub/sub support in the next release of SocketStream (0.4) altogether to allow people to roll their own solutions.

Replicating the current behaviour of multiplexing events would be trivial (4-5 lines of code in your app); but I agree with you that larger installations (which SocketStream is really targeted at) will want to ensure only the bare minimum of traffic flows between Redis and each SocketStream server. The exact implementation of this will depend upon the app's design.

Just a thought at the moment, but I welcome your thoughts.

@timothyjoelwright

@Owen

I've been thinking more about this recently as well. We had decided to be adventurous and use SocketStream for several projects instead of our usual stack. All of these projects require some kind of inter-client communication. The increasing popularity of PubNub and Pusher reinforce that this is a feature that's often at the center of many types of apps. If you decide to remove it, would you stick with a modular approach or would we each write our own app-specific sub/sub implementation? A modular transport-type approach would open up to a lot of options beyond redis: RabbitMQ, AMQP, possibly ZeroMQ/crossroads.IO again.

As a side note we ran load tests with both PubNub, Pusher, and the current SocketStream pub/sub redis implementation. We decided that Pusher and PubNub weren't necessary options since SocketStream managed the loads within 98% of the best performing option (PubNub). The only benefit that any of these services would offer over a standard non-SS app stack with SocketIO would be the decoupling of the event bus from a specific server (At the expense of 2-5x that of hosting the app itself).

@owenb

Interesting findings with regards to PubNub and Pusher!

Right now the pubsub transport in 0.3 is completely modular and the API is frozen, so people could already make a module for RabbitMQ or ZeroMQ or anything else, should they wish.

What I'm really getting at is: is this the best approach?

Maybe for small/medium scale apps, but for really large scale apps (the ones I want to build), I think it would be best to make the app subscribe to individual Redis channels and then use ss.publish.channel() to push these events to the relevant clients in the most efficient way possible.

This would allow you to send messages over Redis (or ActiveMQ, or whatever) in the best message format for your app, which may not even be JSON at all.

It's just a thought at the moment. Need to do more real world testing before I make a decision here. Opinions welcome.

@timothyjoelwright

Ahh ok - I see the direction you're referring to. Let me know if I can be of any help with testing / implementation / etc!

@arxpoetica
SocketStream member

Just my two cents. Removing Redis seems like a big step, so I'd like to hear what you plan on doing.

@ostera

@americanyak
It's more like decoupling than removing.

@arxpoetica
SocketStream member

How so?

@ostera

@americanyak
I'm not going to involve in a debate about word semantics, but it's clear that if a component is removed it's both not included and no longer supported, as in not part at all or completely replaced by some other component or whatnot.

On the other hand, decoupling is about letting a component be itself on its own, allowing it back again whenever one wants to plug it, yet providing the interface for many more custom, new, different or specific components to fill that same gap.

Does that make sense?

@owenb

Exactly. Pretty much decided to remove Redis as a dep now, especially as some folks are having problems deploying their apps to NodeJitsu because of the hiredis dependency.

However, Redis will continue to be a great fit for SocketStream and I will include full instructions on how to use it as a session store or pub/sub transport. The only real difference is that you will add it as a dependency to your app, rather than it being a dependency of SocketStream.

@arxpoetica
SocketStream member

Nice.

@paulbjensen
SocketStream member

This feels like something to be dealt with in a major update to SocketStream. Thus moving to 0.4 milestone.

@paulbjensen paulbjensen added this to the 0.4 milestone
@kulicuu
SocketStream member

Agree this is critical. Another I'm interested in, though quite a queue already.

@arxpoetica
SocketStream member

Completely agree.

As a side note, I've been looking a lot at Sails lately. They've borrowed heavily from Rails, so they get a lot of things right in that they've already had the background to help them get started faster.

I think we're coming from a different paradigm (different needs, different services), but there are a few things that Sails does right (even over Rails) that I think we could look at. One of them is the sort of interoperability/plug-play they give to different db/services.

I'm in favor of decoupling Redis PubSub as a starting point, but simultaneously making official support via a plugin. It's what we were already talking about w/ 0.4, but it seems worth restating. :+1:

@thepian thepian modified the milestone: 0.4.x, 0.4
@thepian thepian modified the milestone: 0.6, 0.4.x
@arxpoetica
SocketStream member

@thepian ^ just bookmarking?

@thepian

Something to compare with that has a very good explanation could be the base for one of the example apps

@thepian thepian modified the milestone: 0.7.0 unstable, 0.6 stable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.