Running dashing under a load balancer

[meatheadmike]

I'm attempting to run dashing-rails under a load balancer and ran in to a couple of issues that I'm hoping you might have some insight in to:

a) How do you keep rufus running jobs on only one box so that it doesn't trigger on each and every machine?

b) Is there an easy way to replace redis with something more stable / scalable? Apparently redis clustering support is a "work in progress" (according to their own site). Additionally it wont work with less than 3 servers. So a mirroring setup is out of the question.

[gottfrois]

Hey, those are tricky and interesting questions. I don't have an answer for a) unfortunately :(

Regarding question b), not sure to understand what kind of issues do you face when using redis. Redis is supposed to scale pretty well so i'm surprised. What would you suggest replacing it by ?

[gottfrois]

Actually a quick thought, we could probably use the AMQP protocol which is often used to publish/subscribe events. This protocol could match our use case here and is I believe much more scalable. Have a look at Hutch

[meatheadmike]

RabbitMQ support would be AWESOME! We already use that quite a bit at our company. So count me in on Hutch. I haven't delved too deeply in to the actual dashing-rails source. Is it easy to set up, or would this be a feature for a future release?

As for the Rufus issue, I was looking at the Rufus docs and it appears they have a Scheduler#shutdown function. So in theory all I'd have to do is make sure it's only running on one box and issue a shutdown upon dashing's init on all the other boxes.

[gottfrois]

It would need to abstract the use of either Redis or RabbitMQ. This would be for a new release. I'll look at the best way to do that. First thought would be to have an abstract class responding to publish and subscribe and then have one interface implementing those methods for each protocol.

There is currently no such thing as you can see here or here.

[gottfrois]

I have just pushed a new branch here with this kind of abstraction. I'm not sure I want to add an extra dependency to the gem by adding Bunny to the gemset.

However with this architecture you should be able to add your own "messenger" (redis by default). I have created the AMQP messenger (not perfect but working) so you can have something to play with.

So basically you just need create the Dashing::Messengers::Bunny class somewhere in your application and add it to dashing.rb initializer like so:

config.messenger_class        = ::Dashing::Messengers::Bunny
config.messenger_url          = 'amqp://guest:guest@localhost:5672/'

I would love to have your thoughts and comments on the code.

Please let me know if you have any questions.

Note that this branch is working but I will probably make changes to it. For example the "messenger" naming is not quite good, maybe "streamer" ? We'll see.

[meatheadmike]

That's awesome! I'll give this a go as soon as I can free up some time :-)

[gottfrois]

Hey man, any news regarding amqp ?

[meatheadmike]

Unfortunately I have been swamped :( I'm hoping to make some headway with this in the next couple of weeks. For the time being I'm making do with redis.

[stancoffyn]

I just made a copy of this branch. fingers-crossed

[gottfrois]

If you guys are interested in the AMQP we could make it a plugin to include in the gemfile

[meatheadmike]

I know I'm definitely interested :-) I'm still fairly green on Ruby/Rails. So anything that makes life easier is greatly appreciated.

[gottfrois]

have you guys got a chance to test that AMQP thing ?

[meatheadmike]

Unfortunately not yet... I do have it on my do-list to play around with though.

[gottfrois]

Closing this issue for now. Please feel free to re-open.