Performance is misleading

[mperham]

Your Performance wiki page is misleading. Benchmarking noops isn't anything close to realistic or useful to potential users. Please provide a real benchmark where each message requires some real processing (hits a database, performs some calculation, etc) and I think you'll find that Sneakers doesn't process jobs any faster than Sidekiq.

[mperham]

I'll get you started:

class EasyProcessor
  include Sneakers::Worker
  from_queue :work

  def work(msg)
    puts msg
    sleep 1
  end
end

class EasyWorker
  include Sidekiq::Worker

  def perform(msg)
    puts msg
    sleep 1
  end
end

[mperham]

I really do wish you luck with Sneakers but remember that your users trust is valuable. Unrealistic benchmarks might get you some immediate PR but hurt that trust long-term. Saying you are 10x faster than Sidekiq is simply not true.

[jondot]

Hi Mike. I'll start by saying I'm a fan of you as a developer and Sidekiq as a tool. Been using Sidekiq in production for years now. I'll take your advice seriously - and I hope the following discussion makes sense.

I'll continue by saying that I hate to reinvent the wheel. It's been stated here with an in-depth explanation https://github.com/jondot/sneakers/wiki/Why-i-built-it

I really, really, wanted to use Sidekiq. Or any wheel that was already invented. 

Eventually I did what every engineer would/should do - measure per my workload, use case, and budget.

I benchmarked against Sidekiq because Sidekiq is my reference benchmark. It is what I start every project with and I love it.

Microbenchmarks are silly. And it's stated at the opening paragraph - but - I will make sure to state it in a more explicit way and explain why it is silly. However, when dealing with infrastructure like I do on a daily basis, I use microbenchmarks to measure my upper bound, happy path of a tool which gives me an initial idea of the plumbing.

What you're saying is completely true - doing long, I/O bound jobs would completely obliterate any performance differences when set up in a similar manner (same size threadpool, same process allocation).

But If you ever need to do very fast jobs (such as event processing, which is one of my use cases), it starts to matter.

Finally, if I included an I/O bound use case for a performance comparison - this would be more misleading because there's no limit to how slow or how complex or how strange your use case is. In some arbitrary cases Sidekiq, or Resque, or delayed_job or even a simple cron job would win over your KPIs.

I will state it explicitly - the best way to benchmark is by oneself - with one's own use case.

As a conclusion I will make the following updates to the Wiki page, which admittedly, was initially written in haste.

• Highlight in a better way the pitfalls of microbenchmarks
• Make it important to understand - when judging a tool one should use his/her own workloads
• Reference this discussion
• Make a note - that for long I/O bound jobs, the only advantage of Sneakers would be by easing the process management for you. And by that fact the factors for deciding what a tool to use fall on different qualifiers.

Sneakers was closed-source for a while, and I've used it for my own benefit for a while. My thinking was to open-source it and share it with others so they could be happy as I am with it.
Before building Sneakers I tried contacting people from Sidekiq and Celluloid community to see how I can take out Redis and replace it with RabbitMQ, but that didn't work as I was being told it isn't a good idea.

In an effort to promote a healthy discussion - I hope to get feedback from you about my suggested alterations if this is acceptable.

Thanks

Edit: I've now completed the updates.

[mperham]

You are marketing Sneakers as a general purpose framework for background jobs and the common case is jobs with I/O, sending email, talking to a database, etc. MRI is not a good VM for CPU intensive tasks so people don't use it for things like real-time stream processing. Your use case might be perfectly reasonable but isn't common in the Ruby world. A benchmark which fakes some I/O via sleep is more general purpose IME.

I have no doubt that Sneakers and RabbitMQ can dispatch more messages per second. Sidekiq has to make 3 roundtrips to Redis for each job, suboptimal but that's the price of using a general purpose tool like Redis.

Thanks for updating the wiki page.

The other thing I'd ask is that you publish the code for the benchmark so others can reproduce your results.

[jondot]

Thanks Mike.

I wasn't intending to market it as general purpose since Sidekiq is already great at that (actually I'm not sure I am even marketing per-se). In a retrospect, analyzing my wording, I guess I hoped attaching "performance" to the tag line would hit the cases where performance-oriented work loads is more important than the general work or all-purpose jobs, for example - job profiles with low-latency and high req/s such as event and log processing based scenarios.

i.e. perhaps the difference is better illustrated here:

all purpose background-processing framework for Ruby (Sidekiq)
vs.
performance background-processing framework for Ruby (Sneakers)

Seemed to me, at the time, that picking the second one focused my library at those niche cases where you need high-performance job profiles to work.

Obviously, I use both Sidekiq and Sneakers to cover both those cases in production myself.

Regarding stream processing on MRI, might be beneficial to check out fluentd http://fluentd.org/, which was good enough for me up to a point where I needed to develop the concept of 'worker' that can be maintained independently; I also didn't need the extra fluff of general stream processing, and wanted to have the option to have control over I/O performance for those cases where I needed to push events through I/O and more AMQP flexibility and semantics introduced into workers.

Anecdotally, Sneakers is based on server_engine, the process management core extracted out of fluentd.

[Donavan]

Wouldn't a benchmark that includes sleeps just add noise? "Oh look everything is one second slower now that we added a one second sleep". I care about the overhead on the framework the IO time isn't going to suddenly change because my job is in sneakers now...

On Jan 10, 2014, at 11:49 AM, Mike Perham notifications@github.com wrote:

You are marketing Sneakers as a general purpose framework for background jobs and the common case is jobs with I/O, sending email, talking to a database, etc. MRI is not a good VM for CPU intensive tasks so people don't use it for things like real-time stream processing. Your use case might be perfectly reasonable but isn't common in the Ruby world. A benchmark which fakes some I/O via sleep is more general purpose IME.

I have no doubt that Sneakers and RabbitMQ can dispatch more messages per second. Sidekiq has to make 3 roundtrips to Redis for each job, suboptimal but that's the price of using a general purpose tool like Redis.

Thanks for updating the wiki page.

The other thing I'd ask is that you publish the code for the benchmark so others can reproduce your results.

—
Reply to this email directly or view it on GitHub.

[mperham]

@Donavan Do your jobs take no time to process? If Sneakers takes 50 usec of overhead and Sidekiq takes 100 usec of overhead but your job takes one second to actually process, you are talking about a miniscule difference in real world performance.

If your jobs take 1 ms to process, the overhead becomes relevant but most people don't have jobs that take 1 ms to process.