ActiveJob Memory bloat

[inkstak]

Hi, I'm investigating on a memory bloat linked to ActiveJob.
I first reported it to Sidekiq but the problem seems to come from ActiveJob.

System configuration

Rails 5.0.0 (also reproduced with master)
Ruby 2.3.1

Steps to reproduce

An app to reproduce is available here.
Setup the database to create thousands of records.

rails db:setup

The job MemTestJob requests the records by batches, call garbage collector and output the heap & memory usage.

In a console, run the job:

$ MemTestJob.perform_now

GC.heap_available_slots: 1,488,576
GC.heap_live_slots: 167,099
GC.heap_free_slots: 1,321,477
RSS: 205 MB
[...]
GC.heap_available_slots: 1,224,441
GC.heap_live_slots: 178,136
GC.heap_free_slots: 1,046,305
RSS: 205 MB
[...]
GC.heap_available_slots: 1,157,178
GC.heap_live_slots: 178,136
GC.heap_free_slots: 979,042
RSS: 205 MB

The memory usage and heap size will quickly stabilize.

Now, run the job as ActiveJob do asynchronously:

$ ActiveJob::Base.execute("job_class" => "MemTestJob", "arguments" => [])`

GC.heap_available_slots: 273,908
GC.heap_live_slots: 166,897
GC.heap_free_slots: 107,011
RSS: 61.1 MB
[...]
GC.heap_available_slots: 300,401
GC.heap_live_slots: 206,301
GC.heap_free_slots: 94,100
RSS: 67.8 MB
[...]
GC.heap_available_slots: 302,024
GC.heap_live_slots: 236,421
GC.heap_free_slots: 65,603
RSS: 69.7 MB

The heap size and the RSS increase each round until the final blow up.
In this example, it grows a few bytes per request, but it represents several MB on a real apps with many gems.

Investigation

After digging the code, the cause seems to be linked to the reloader in ActiveJob

The increase of memory may be handled in development, but is problematic in production when browsing very large databases. Can we skip this callback when the config.eager_load is set to true ?

[maclover7]

cc @matthewd, since this has to do with ActiveSupport::Reloader

[eugeneius]

This is caused by the query cache, which is enabled by the reloader.

If you run the job directly with the cache enabled, you'll see memory usage grow steadily:

ActiveRecord::Base.connection.cache { MemTestJob.perform_now }

Conversely, if you disable the cache while the job is running, memory growth will taper off:

def perform
  ActiveRecord::Base.connection.uncached do
    Item.find_in_batches.with_index do |_, index|
      gc_start if index % 10 == 0
    end
  end
end

This behaviour is intentional - Rails is using more memory to avoid duplicate queries. The cache is cleared when the job finishes, so memory isn't being permanently leaked, but peak usage will be higher. The cache is also cleared when the connection writes to the database - in a real app, you would presumably write at least once per batch, and simulating this also controls memory usage:

def perform
  Item.find_in_batches.with_index do |batch, index|
    gc_start if index % 10 == 0
    batch.first.touch
  end
end

The example job here is a pathological case, where a million rows are read but nothing is written. While the query cache clearly doesn't help in this particular scenario, I'm not sure if there's anything we can do here, other than to recommend turning it off in these extreme cases.

[inkstak]

Thanks for the explanation and the workaround.
I use now the Item.uncached method to prevent memory growth.

One real-life example is indexing a large database : it read millions of rows without write.

Anyway, is it really necessary to cache by default batched queries ? Batches are intended to loop through a large collection of records, whatever you'll write or not, when a each is not efficient.
Each database queries should be unique unless you expect to run the batches twice.

[matthewd]

I'll close this issue, as it was focused on Active Job... but @inkstak if you'd like to open a new issue that batches should skip the query cache (or a PR 😁), I think that's a reasonable point.