Why i built it
Truth is, I've already built a background processing framework specifically on JRuby, called frenzy_bunnies.
It uses the top-notch Java RabbitMQ driver (wrapped by the JRuby library HotBunnies), and it drills down to "bare-metal" Java for performance which made it rock-solid in production for the last 1.5 years, processing millions of I/O and computationally intensive jobs. All under negligible CPU and memory usage (~200MB when most busy).
It's been running in production, requires zero-maintenance. If you're on JRuby, you can use it and stop here :)
CRuby, Performance, and High-Availability
Then came a project where I needed really good velocity, to appeal to non-JRuby/Java devs, and to have really good performance. This time, I also didn't have the luxury of finding replacements in JRuby-land for gems using c-extensions should I bump into them.
I really, really, wanted to use Sidekiq. Or any wheel that was already invented. But I was getting held back at 600req/s on an EC2-Large (a microbenchmark with empty jobs) and I felt a good starting point would be to check off that limit.
Nor could I allow the single-point-of-failure that being Redis, which isn't really suitable for a highly-available background processing framework - I couldn't lose messages (it's worth mentioning that Redis is often my go-to swiss-army knife and you'll have to pry it from my dying corpse).
Compared to Sidekiq, per my use case, I needed
- A great-performing framework limited only by broker speed - at least 1000req/s acknowledged and persisted on EC2-Large (Sneakers does more than that), and
- That would use all cores
- A highly available processing framework (here we have same guarantees as RabbitMQ offers, which is great)
- A familiar DSL/API that also supports advanced messaging semantics such as reject, requeue, acknowledge, etc, and
- That would not expose the whole guts of AMQP at me, but just-enough from it.
And irrelevant of the comparison to Sidekiq or any other background processing framework I needed
- It should use a ruby that doesn't care about content of gems and can run C-extensions. MRI.
- A production-ready package that holds all of these together allowing me to be as lazy as possible
- Metrics and logging baked in
- Convenient deployment, maintenance and supervision story