YJIT is indirectly retaining ActiveRecord Object instances

[intrip]

We noticed a slow and steady memory leak in our web app, to investigate it I enabled ObjectSpace.trace_object_allocations_start and extracted a few heap dumps, then started analysing them with sheap.

I found a common pattern: A yjit_root node holding references to ActiveRecord methods holding references to instances of ActiveRecord models then holding references to their instance attributes. These objects attributes sums up and creates a slow and steady memory leak:

[<ROOT vm (3194 refs)>,
 <DATA 0x7fc02024b070 yjit_root (11268 refs)>,
 <IMEMO 0x7fbffcfa5370 ment (4 refs)>,
 <IMEMO 0x7fc01d729090 iseq (491 refs)>,
 <OBJECT 0x7fc01d43c9e0 (0x7fc01d43c800) (6 refs)>,
 <OBJECT 0x7fc01d466740 ActiveRecord::Associations::HasManyThroughAssociation (7 refs)>,
 <OBJECT 0x7fbfd60d1108 Entry (6 refs)>,
 <HASH 0x7fc0203c6198 (11 refs)>,
 <OBJECT 0x7fc01c866418 ActiveRecord::Associations::BelongsToPolymorphicAssociation (5 refs)>,
 <OBJECT 0x7fbfd5f74f58 Message (3 refs)>,
 <HASH 0x7fc01c6feda0 (1 refs)>,
 <OBJECT 0x7fbfe5875af8 ActiveRecord::Associations::HasOneAssociation (4 refs)>,
 <OBJECT 0x7fbfd5f622b8 ActionText::EncryptedRichText (3 refs)>,
 <OBJECT 0x7fbfe5871a98 ActiveModel::LazyAttributeSet (6 refs)>,
 <HASH 0x7fc01c6f9300 (5 refs)>,
 <OBJECT 0x7fbfd5f61228 ActionText::Content (6 refs)>,
 <OBJECT 0x7fbfd5f3da08 ActionText::Fragment (1 refs)>,
 <OBJECT 0x7fbfe58c2948 Okra::HTML::Node (1 refs)>,
 <ARRAY 0x7fbfd5f611d8 (3 refs)>,
 <OBJECT 0x7fbfe58c2998 Okra::HTML::Node (3 refs)>,
 <ARRAY 0x7fbfe5871458 (5 refs)>,
 <OBJECT 0x7fbfe58c2c68 Okra::HTML::Node (3 refs)>,
 <ARRAY 0x7fc01f2a0008 (55 refs)>,
 <OBJECT 0x7fbfe58ab8b0 Okra::HTML::Node (3 refs)>,
 <ARRAY 0x7fbfe58ab9a0 (8 refs)>,
 <ARRAY 0x7fbfd5f5c390 (2 refs)>,
 <STRING 0x7fbfd5f5c3b8 "630">]

This is the node data until you reach the ActiveRecord model instance:

irb#1():068> $diff.after.at("0x7fbffcfa5370").data
=>
{"address"=>"0x7fbffcfa5370",
 "type"=>"IMEMO",
 "shape_id"=>0,
 "slot_size"=>40,
 "imemo_type"=>"ment",
 "references"=>["0x7fc01d578660", "0x7fc01d5d6c38", "0x7fc01d729090", "0x7fc01d854050"],
 "file"=>"/usr/local/bundle/ruby/3.3.0/bundler/gems/rails-139c5678aa5d/activerecord/lib/active_record/relation/query_methods.rb",
 "line"=>1595,
 "method"=>"build_arel",
 "generation"=>18,
 "memsize"=>48,
 "flags"=>{"wb_protected"=>true, "old"=>true, "uncollectible"=>true, "marked"=>true}}
irb#1():069> $diff.after.at("0x7fbffcfa5370").data.except("references")
irb#1():073> $diff.after.at("0x7fc01d729090").data.except("references")
=>
{"address"=>"0x7fc01d729090",
 "type"=>"IMEMO",
 "shape_id"=>0,
 "slot_size"=>40,
 "imemo_type"=>"iseq",
 "file"=>"/usr/local/bundle/ruby/3.3.0/bundler/gems/rails-139c5678aa5d/activerecord/lib/active_record/relation/query_methods.rb",
 "generation"=>4,
 "memsize"=>1736,
 "flags"=>{"wb_protected"=>true, "old"=>true, "uncollectible"=>true, "marked"=>true}}=>
{"address"=>"0x7fc01d43c9e0",
 "type"=>"OBJECT",
 "shape_id"=>5394,
 "slot_size"=>160,
 "class"=>"0x7fc01d43c800",
 "embedded"=>true,
 "ivars"=>18,
 "references"=>["0x7fc000f4c000", "0x7fc007f63168", "0x7fc01d43c760", "0x7fbfec306138", "0x7fc01d466740", "0x7fbfd5fd30f8"],
 "file"=>"<internal:kernel>",
 "line"=>48,
 "method"=>"clone",
 "generation"=>31,
 "memsize"=>160,
 "flags"=>{"wb_protected"=>true, "old"=>true, "uncollectible"=>true, "marked"=>true}}
irb#1():072> $diff.after.at("0x7fc01d466740").data
=>
{"address"=>"0x7fc01d466740",
 "type"=>"OBJECT",
 "shape_id"=>5439,
 "slot_size"=>160,
 "class"=>"0x7fbfed5b4ed8",
 "embedded"=>true,
 "ivars"=>12,
 "references"=>["0x7fc01c55e108", "0x7fbfd60d1108", "0x7fbfd5fdad30", "0x7fbfd5fdad08", "0x7fc01d463b80", "0x7fc01d466560", "0x7fc01d466380"],
 "file"=>"/usr/local/bundle/ruby/3.3.0/bundler/gems/rails-139c5678aa5d/activerecord/lib/active_record/associations.rb",
 "line"=>320,
 "method"=>"new",
 "generation"=>31,
 "memsize"=>160,
 "flags"=>{"wb_protected"=>true, "old"=>true, "uncollectible"=>true, "marked"=>true}}
irb#1():073>
irb#1():075> $diff.after.at("0x7fbfd60d1108").data
=>
{"address"=>"0x7fbfd60d1108",
 "type"=>"OBJECT",
 "shape_id"=>6155,
 "slot_size"=>40,
 "class"=>"0x7fbfed48ed88",
 "ivars"=>26,
 "references"=>["0x7fbfdee4b420", "0x7fc01e29a820", "0x7fc0203c6198", "0x7fbfecef2820", "0x7fbfeceef0a8", "0x7fbfecee2038"],
 "file"=>"/usr/local/bundle/ruby/3.3.0/bundler/gems/rails-139c5678aa5d/activerecord/lib/active_record/persistence.rb",
 "line"=>642,
 "method"=>"allocate",
 "generation"=>31,
 "memsize"=>296,
 "flags"=>{"wb_protected"=>true, "old"=>true, "uncollectible"=>true, "marked"=>true}}

The fact that ActiveRecord instances hold references to their data looks good to me, to me looks like the issue is that yjit_root keeps indirectly referencing them, which prevents the objects from being GCed hence the leak.
I've tried disabling YJIT on 1 host (hey-default-app-109) and the leak was gone, you can see it from this overnight graph of memory usage in the hosts:

In these hosts we are running YJIT with the default options so just RUBYOPT="--yjit"

Can you help me figure out what's going on here? This issue applies only to a subset of web hosts, which receives specific traffic that creates the "Entry" > "Message" > "Okra" objects.

Happy to tell you any additional info that could help, the next week I'll be OFF so if I won't answer quickly that's the reason 😅. Thanks!

[maximecb]

Hi @intrip,

Thanks for the bug report. I assume this is running on Ruby 3.3.0 ? IIRC we may have had another report of a memory leak.

@XrXr @k0kubun I'm wondering if this might be happening because we root some objects in YJIT when we bake addresses in the machine code, and we now have code GC disabled, so even dead ISEQs could still have code that never gets collected?

[maximecb]

Might be useful if we could come up with a script that reproduces this memory leak to make it easier to debug. Maybe something that creates ActiveRecord models in a loop?

[intrip]

Hi @maximecb,

Thanks for the bug report. I assume this is running on Ruby 3.3.0 ? IIRC we may have had another report of a memory leak.

Yes, it's Ruby 3.3.0.

To workaround it for now I've reduced the YJIT memory to 32Mib and that mitigated greatly the issue, but it dropped a bit the ratio_in_yjit (from 98 to 95). I've also noticed that enabling GCs in YJIT mitigated it but led to worse overall performance so for now I've just reduced the overall memory.

We noticed it now because it became worse recently (not sure if it's related to the Ruby upgrade though) but TBH it's been a while that we were affected by this issue, at least 90+ days.

Might be useful if we could come up with a script that reproduces this memory leak to make it easier to debug. Maybe something that creates ActiveRecord models in a loop?

I'll give it a try but I'm about to leave now and I'll be OFF the next week so it might take some time sorry 😅.

[maximecb]

No problem. I wish we could fix the issue faster :)

[casperisfine]

@intrip do you make any use of Relation#extending? Or other ways to extend a module in relations?

[casperisfine]

Also, I think if you were to strip all the T_STRING value fields, you should be able to share the dump without leaking any personal data. Either publicly or in direct to one of us, that would be helpful to let us dig in the heap dump.

[casperisfine]

Might be useful if we could come up with a script that reproduces this memory leak to make it easier to debug.

I suspect it's the same bug than https://bugs.ruby-lang.org/issues/19436, but with YJIT (and worse because the references accumulate instead of only retaining the last one).

[maximecb]

I just remembered that Aaron @tenderworks had mentioned a memory leak with YJIT 3.3 and Mastodon. This may be related. Would be good if we could get more info about the Mastodon issue.

[k0kubun]

I'm wondering if this might be happening because we root some objects in YJIT when we bake addresses in the machine code, and we now have code GC disabled, so even dead ISEQs could still have code that never gets collected?

They've been running 3.3.0-preview3 since November #548 (comment), which had code GC disabled unlike preview2. So I'm not sure if it explains the situation.

Also, Blocks are deallocated when the ISEQ is GCed (along with their gc_obj_offsets), so "dead" ISEQs would not mark baked objects whether Code GC is disabled or not.

Anyway, a local repro would be really helpful for identifying the cause.

[casperisfine]

I suspect it's the same bug than https://bugs.ruby-lang.org/issues/19436, but with YJIT

Alright, that's what I suspected:

module Foo
  def bar
  end
end

def call_bar(obj)
  # Here the call cache we'll keep a ref on the method_entry
  # which then keep a ref on the singleton_class, making that
  # instance immortal until the method is called again with
  # another instance.

  # The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT
  obj.bar
end

obj = Object.new
obj.extend(Foo)

5.times do
  call_bar(obj)
end

id = obj.object_id
obj = nil

4.times { GC.start }
begin
  ObjectSpace._id2ref(id)
  puts "MEMORY LEAK!!!"
  exit!(1)
rescue RangeError
  puts "no leak"
  exit!(0)
end

$ ruby -v /tmp/repro.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
no leak
$ ruby -v /tmp/repro.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
no leak
$ ruby -v /tmp/repro.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
no leak
$ ruby --yjit --yjit-call-threshold=1 -v /tmp/repro.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) +YJIT [arm64-darwin23]
MEMORY LEAK!!!
$ ruby --yjit --yjit-call-threshold=1 -v /tmp/repro.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) +YJIT [arm64-darwin23]
MEMORY LEAK!!!
$ ruby --yjit --yjit-call-threshold=1 -v /tmp/repro.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) +YJIT [arm64-darwin23]
MEMORY LEAK!!!

So @intrip is very likely using Relation#extending, which is a giant footgun and we really should deprecate that API in Rails (Ref: rails/rails#47314)

[maximecb]

Following up on the discussions we had in meetings. Jean said that call caches must be weak references. @XrXr do we root objects / CMEs when compiling Ruby calls in YJIT? If so, could we make them weak references?

[XrXr]

We do root CMEs because we root everything we refer to from generated code. Making them weak references will take some effort, though. Even with a constant pool we can't directly make use of the current mechanism that sets the VALUE to Qundef when the referent dies, because we have a GC yield point before writing out CMEs. If the CME gets collected during the GC we end up pushing a corrupted frame.

[byroot]

Would not compiling singleton instance methods be an easy workaround? Their perf already suck on the interpreter, so doubt it would impact YJIT performance noticeably.

And these methods are very likely not to survive long anyway, so compiling them is a bit of a waste.