Join GitHub today
Memory leak with Sidekiq 5. #3824
I'm running into a memory leak (Sidekiq on Heroku) where the sidekiq process memory grows to more than 1 GB.
I can also reproduce the issue locally on my machine.
I'm using ActiveRecord so I have tried cleaning the cache at the end of every job, but that didn't help.
I have removed everything from my worker (an empty
I have used memory_profiler (as setup below in the Sidekiq initializer) to try to find out what the leak could be but I don't see anything suspicious and certainly not hundreds of megabytes of leaked data:
This points to me to, maybe, a leak in a C extension but at this point I would be interested in getting some advice on how to proceed forward. I'm not sure there are any C extensions involved in running a sidekiq process? Or maybe in the redis client?
Here is also my Gemfile.lock:
Below is the requested information about Sidekiq.
Thanks for any advice!
Ruby version: 2.4.4
Please include your initializer and any error message with the full backtrace.
Are you using an old version? No
Have you checked the changelogs to see if your issue has been fixed in a later version? Yes, but I'm running the latest version.
I have no idea what the leak might be. Sidekiq is pure Ruby but you have many native gems that could have issues. There are many other things in MRI that can cause excessive memory usage:
Thanks for your reply.
The problem appeared when switching to Sidekiq 5/Ruby 2.4 from Sidekiq 3/Ruby 2.1.4.
I'm not sure other gems would be responsible for this issue:
I will investigate the MRI part but I'm using Ruby 2.4.4 which is fairly "standard" and I would have expected to see this problem in other places.
For now it seems that reducing the Sidekiq concurrency from 25 to 5 seem to have helped contain the memory growth.
However, I don't understand why because while we had 25 threads for Sidekiq, we never have more than 2-3 jobs running at the same time and as far as I understand, a Sidekiq thread will not require much memory by itself and should share most of the memory with the other threads. So it seems that just having the ability to dispatch to more threads leads to an increase in memory.
@mperham problem solved. Thanks for pointing us in the right direction.
The initial issue wasn't really a leak but just a memory bloat (meaning that too much memory was used) as you had suggested.
So that this might help others:
The issue appeared when upgrading Heroku from cedar-14 to heroku-16. We had to bump Ruby from 2.1 to 2.4. We also moved from Sidekiq 3 to Sidekiq 5. Lots of moving pieces so hard to pinpoint the issue.
First we checked that there was no leak using the
We reduced the concurrency of Sidekiq from 25 to 5 and this clearly helped reduce the memory usage considerably even though we are not running at full concurrency at all (most of the time we only have a couple jobs running). But we could still see the memory slowly increase over time. Just "having" the threads by themselves seemed to have created a memory bloat even though they were not active at the same time; this is a bit counterintuitive because you would think Ruby requests more memory when two threads request are active at the same time and increase the contention.
Then we read the article you suggested above https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html and it made entirely sense with the behavior we were seeing: not a memory leak but slow memory increase because of memory fragmentation.
We decided to switch the memory allocator to
It's worth investing some time in understanding this issue, but in the end the solution is very simple and doesn't require anything complex.
For anyone coming across this thread. I would recommend upgrading Ruby versions separately from migrating stacks, so that way it's easier to isolate if memory growth (or any other issues) happen due to the version changes or the stack change.
@bvirlet thanks for the detailed write up. Glad you got it working.