High memory consumption using Sidekiq with ActiveJob

[davidpiegza]

Ruby version: 2.4.3
Sidekiq / Pro / Enterprise version(s): 5.1.1

I discovered a high memory usage when using Sidekiq together with ActiveJob. I created a sample app at davidpiegza/sidekiq-memory-test#1.

In my tests, the results for creating 250_000 jobs are:

With ActiveJob: 520 MB
Without ActiveJob: 60 MB

The tests run on a new created Rails app in production environment with a sidekiq concurrency of 10. This high memory consumption also happens on Heroku with multiple workers running.

[mperham]

In designing and building Sidekiq, I've tried to make pragmatic trade-offs between performance and developer friendliness. I can also optimize the happy path and provide special APIs for edge cases. For instance, you can make your benchmark even faster if you use the push_bulk API. This load tester creates 100,000 native jobs in 2 seconds (vs 40 sec for AJ) on my laptop:

https://github.com/mperham/sidekiq/blob/master/bin/sidekiqload#L92-L100

AJ has much more complex Ruby serialization/deserialization logic, which will also chew up memory/CPU. Sidekiq's de/serialization is explicitly "JSON only" (one of those pragmatic decisions mentioned above) and therefore is implemented in pure C within the "json" gem.

And when you actually want to process those jobs? The 100,000 jobs take 22 seconds to process on my laptop. The same AJ jobs take 65 seconds to process, so ~3x the overhead.

[elia]

Having heard from many people that is crucial not to use Sidekiq through AJ for performance reasons I was wondering about the absolute numbers of the overhead difference. I suspected that it might be not so relevant for many use-cases in which the time taken for performing the job is orders of magnitude more than the overhead of AJ.

Running the numbers provided by the example turns out that (65.0 / 100_000.0 - 22.0 / 100_000.0) results in about 0.43ms per job.

Since many many jobs are extracted to perform slow operation out of the request-response cycle I suggest adding a note to clarify that the 3x difference is relevant only for super-large amounts of jobs that execute very fast. ⚡️

@mperham performance numbers are tricky, let me know if I missed something in this line of reasoning or if you have any other thoughts/experience on this topic 🙏

[mperham]

@elia I was addressing the creator's issue at hand. I don't really see the point of people jumping into a discussion years later to muddy the waters. You are right that performance and benchmarks are always highly contextual. Put up a blog post if you want to educate the broader community.