I love this story, and the powers that be. And, another reason to brew resque.

very slick

is there by any chance a possibility that you are going to write up something how that whole rs.github.com schema / cluster accessing works? would be awesome, just like your other architecture stuff :)

and don't forget about storage! :-)

dang. http://aux1.rs.github.com:9292/resque/overview times out. Would be nice to see how things are going (users can answer there own slowness issues, i.e. my commit isn't showing -> 10th in the queue ^_^ ). I assume the whole interface is read only, that you control it by other means....

I can't help but shake this nagging feeling that you're overloading the list feature of Redis by using it as the basis for a queue of this scale. I just evaluated Recue (and thus Redis) as a replacement for a basic in-house queue we're using.. the problem I have is that workers have to poll the Redis list and that screams to me that ideally what you want there is async message retrieval from something like an AMQP broker. Obviously the sum benefits Redis provides over using an AMQP broker is the basis for Resque existing (ability to easily store state etc), but if Resque were a mix of Redis (for worker state) & AMQP (for job processing) your throughput would be higher and you wouldn't need to run as many workers.

Would you consider a pluggable system for the job processing portion of Resque? I don't see a reason why Resque couldn't use AMQP for job retrieval and still use Redis for Stat and working_on? Let me know what you think, I'll have a go at implementing it if you're not completely opposed to the idea.

This is just lovely :)

@ileitch: I love Redis, but I get the same felling also.

I use Gearman internally, and I've also used AMQP solutions in the past, and I think Gearman would be a much better fit.

But as long as it works for them, and keeps my repos available, its all good.

I like the work you guys are doing on ernie, bert-rpc and resque. How do they fit together?