Aliasing a method from a class in oldgen on a short-lived singleton class that is collected at the next GC causes a segfault via a type confusion #5534
Description
I've been investigating a reproducible segfault. My findings are here: artichoke/artichoke#1327 (comment).
Unfortunately, I do not have a minimized reproduction.
Dataflow
I think this is what's happening:
-
A
-@method is defined in Ruby for theNumericclass. -
Numericis moved to the old generation. -
Sometime later and all before the subsequent GC:
- An object
objis created from a subclass ofNumeric. obj.respond_to? :-@is called which pulls the proc forNumeric#-@into the method cache.obj.singleton_class.alias_method :some_other_symbol, :-@is called, which converts theRProc *forNumeric#-@to haveebe anREnvvariant. When attaching the proc toobj's singleton class, a write barrier is called on the newly allocatedREnv *.
- An object
-
Then the object and its singleton class become unreachable.
-
Then a minor GC occurs:
- Because the GC is minor, a
root_scan_phasedoes not occur, which meansNumericand theRProc *s in its method table are not marked. - This causes the
REnv *from the prior alias call to not be painted black. - The
REnv *is freed. - The
REnv *'sRBasic *is returned to the freelist in the GC heap to be reused.
- Because the GC is minor, a
-
Sometime later but before the next full GC, the
RBasic *is reallocated as aRProc*.- This means the
proc->efield on theNumeric#-@proc is now anRProc *instead of the expectedREnv *.
- This means the
-
The newly allocated proc is then involved in an
mrb_alias_methodcall which writes toe->midand corrupts theNumeric#-@Proc'senv. -
When a full GC finally occurs, the
RProc's in theNumericmethod table are added to the gray list. When the GC attempts to mark theNumeric#-@proc'sREnvingc_mark_children, a type confusion occurs causing an invalid memory access.
Proposed fix
When aliasing a method, the RProc's target class should be marked with write barrier so it is used for marking in the next GC.