gc(layer 3): from-space quarantine catches 55 stale dereferences across the gap suite — the instrument is in CI but aimed at one synthetic fixture

[proggeramlug]

Summary

PERRY_GC_PROTECT_FROMSPACE catches 55 stale from-space dereferences across
the 479-test gap suite — 44 of them in programs that otherwise exit cleanly and
produce correct output. The instrument already exists, is well built, and is
already wired into CI. It is simply pointed at the wrong workload, so none of
these have ever been visible.

This is layer 3 of docs/engine-plan.md (rooting in hand-written runtime Rust),
the layer that #7280's surviving fault moved into and that no static gate can
see.

Why these were invisible

scripts/gc_instrument_smoke.sh is a good gate — it asserts both knob
directions and has a real non-vacuity check (zeal must produce strictly more
retirements than no-zeal). But it drives the instrument with:

PERRY_GC_MOVING_LOOP_POLLS=1 ... $WORK/fixture

over one synthetic fixture. Back-edge polls fire only while user JS runs, so
that configuration structurally cannot expose an unrooted pointer in runtime code
that does not re-enter user JS — which is most of the runtime. That limitation is
already documented from #7249; the smoke script predates applying it here.

Driving the same instrument by allocation-point pressure over real
programs instead:

PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64 \
PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off \
  ./test_binary

Results

Every gap test compiled and run twice — quarantine OFF as a control, quarantine
ON — so a pre-existing crash cannot be miscounted as a catch.

measured                                     479
QUARANTINE-ONLY FAULTS (control ok, exp SIGBUS)  55
  ...of which the control exited 0 (clean)       44
faulted in BOTH arms (pre-existing)               9
no fault in either arm                          415

The faults are self-diagnosing:

[gc-fromspace-protect] FAULT: signal 10 at 0x5cb232a03a9
  This address is RETIRED FROM-SPACE. The evacuating minor moved or
  freed the object here and the holder kept the pre-collection address.
  block=0x5cb232a0000 +937 retired_bytes=1048536 retired_by_minor=#0
  last-known object: user_ptr=0x5cb232a03b0 obj_type=1 size=8208

This is not 55 independent bugs

The signatures are heavily concentrated, which is the useful part:

dimension	distribution
collection	49 of 57 are retired_by_minor=#0 — the first minor, 7 at #1, 1 at #2
object type	34 are GC_TYPE_STRING, 8 GC_TYPE_PROMISE, 4 GC_TYPE_OBJECT, 2 GC_TYPE_CLOSURE
size	size=32 dominates (22)

A stale *mut StringHeader across the first minor collection is the dominant
shape. That is consistent with the runtime exposing 218 functions returning
*mut StringHeader with nothing obliging a caller to root the result, and with
#7247 (js_regexp_new holding &str borrows into movable StringHeaders).

The minor-#0 concentration also rhymes with #7217, where the first collection
landed inside a bootstrap holding a raw object graph — worth checking whether
these share that cause before treating them as separate.

Three of the catches are tests named for GC rooting, which is a useful
sanity check that the instrument is finding the right class:
test_gap_gc_assign_string_source_rooting, test_gap_gc_pointer_publish,
test_gap_gc_process_env_cache_rooting.

Verification

• test_gap_gc_pointer_publish faults 20/20 runs — deterministic, not a
  schedule lottery.
• Cross-checked against PERRY_GC_PROTECT_FROMSPACE=poison, which poisons
  retired bytes without changing page protection. Several tests change
  observable behaviour under poison alone (test_gap_json_advanced goes
  rc=0 → rc=1), which confirms retired memory is genuinely being read rather
  than the mprotect being over-eager.

Proposed work

1. Triage the minor-#0 / GC_TYPE_STRING cluster first — it is very likely a
   small number of causes behind ~60% of the catches.
2. Add a CI arm that runs the quarantine over the gap suite by the
   allocation-point route, alongside the existing loop-polls smoke. The
   instrument is already correct; only its aim needs fixing.

Full list of the 44 clean-program catches:

• test_gap_6300_const_cast_assign
• test_gap_6375_legacy_url_parse_format
• test_gap_6386_dataview_concat_regex_fastpaths
• test_gap_6812_object_write_loop_generalization
• test_gap_asynchooks_3089_3090_3091_3093
• test_gap_bind_replace_2840_2867
• test_gap_class_forward_capture_6523
• test_gap_computed_namespace_dispatch
• test_gap_concise_method_explicit_this
• test_gap_cp_3079plus
• test_gap_crypto_cipher_3382_3381_2954
• test_gap_crypto_state_2944_2945_2962
• test_gap_diagchannel_3082_3084_3085_3086
• test_gap_dynamic_import_init_time
• test_gap_encoding_timers
• test_gap_fetch_response
• test_gap_field_lane_semantics
• test_gap_fs_write_error_propagation
• test_gap_gc_assign_string_source_rooting
• test_gap_gc_pointer_publish
• test_gap_gc_process_env_cache_rooting
• test_gap_http2_settings
• test_gap_in_operator_numeric_key
• test_gap_in_operator_to_property_key
• test_gap_json_advanced
• test_gap_json_array_element_overflow_fields
• test_gap_node_crypto_buffer
• test_gap_node_fs
• test_gap_node_process
• test_gap_object_assign_collection
• test_gap_process_misc_3045plus
• test_gap_put_value_plan_cache
• test_gap_readable_stream_tee_pull
• test_gap_reflect_semantics2
• test_gap_repsel_canonical_str_locals
• test_gap_request_binary_body_5483
• test_gap_stream_async_transform_tick_parity
• test_gap_stream_id_band_dynamic_dispatch
• test_gap_stream_livetee_tick_parity
• test_gap_stream_tee_tick_parity
• test_gap_string_index_get_object_receiver
• test_gap_transform_stream_deferred_flush
• test_gap_worker_channels_3157plus
• test_gap_zlib_fs_assert_2935_2752_2971

[proggeramlug]

Post-fix measurement: 55 → 54, and the remaining cluster is sharper

#7342 fixed the array-store-receiver cluster. Re-running the same two-arm sweep
against it:

                        before   after
quarantine-only faults      55      54
faulted in both arms         9       9
clean                      415     416
correctness (vs node)   447/19/13 → 447/19/13   (unchanged, zero regressions)

So that fix closed exactly one test (test_gap_gc_pointer_publish) and
introduced none. Worth stating plainly: the array-store receiver was a real bug
with a deterministic reproducer, but it is a small slice of this issue.

The remaining 54 are dominated by one shape

dimension	distribution
collection	48 of 56 at retired_by_minor=#0, 7 at #1, 1 at #2
object type	41 GC_TYPE_STRING, 8 GC_TYPE_PROMISE, 5 GC_TYPE_OBJECT, 2 GC_TYPE_CLOSURE

A worked example, because it redirects the search

Smallest string-cluster case is test_gap_object_assign_collection (20 lines).

frame #0: js_array_length + 468
frame #1: js_object_assign_one + 1456
frame #2: main + 2084

The interesting part is that js_object_assign_one is already carefully
rooted — #7200 gave it a RuntimeHandleScope, roots target, src, source
and arr, and reads each back. The array branch does:

let arr_h = scope.root_raw_const_ptr(src_raw as *const ArrayHeader);
let arr = arr_h.get_raw_const_ptr::<ArrayHeader>();
let n = crate::array::js_array_length(arr);   // ← faults here

There is no collection point between the root and the read-back, so arr can
only be stale if src_raw was already stale when the scope was opened — i.e.
the argument arrived stale. The last line of that test is
Object.assign({ a: 1 }, { b: 2 }, [10, 20]): the array literal is allocated,
then further allocation happens before the helper is called.

That means a meaningful part of this issue is not layer 3 at all. Rooting a
pointer that is already dangling publishes the dangling value — the fix has to be
at the call site, in the same family as #6969/#7114 argument rooting and #7342's
store receiver, not inside the runtime helper.

Anyone picking this up: get the backtrace first
(--debug-symbols + the allocation-point route below), and check whether the
faulting helper already has a scope. If it does, the bug is in its caller.

PERRY_GC_PROTECT_FROMSPACE=1 PERRY_GC_PROTECT_FROMSPACE_DEPTH=64 \
PERRY_GC_HEAP_LIMIT=8 PERRY_GC_INCREMENTAL=0 PERRY_CONSERVATIVE_STACK_SCAN=off \
  ./binary

#7342 also adds arm 4 to gc_instrument_smoke.sh, which runs this route over the
gc_ratchet probes — so anything reproduced as a probe is gated from then on.

[proggeramlug]

Cluster breakdown of the remaining catches, by faulting site

Backtraced all 31 catches with --debug-symbols and grouped by frame#0 ← frame#1. They are not 31 bugs:

count	faulting site	status
8	DirectParser::parse_value ← js_json_parse	✅ fixed in #7373
5	lookup_fancy_regex ← 4 different callers	caller-side, see below
4	js_promise_state ← main
3	js_symbol_to_string ← js_jsvalue_to_string
3	js_object_literal_infer_computed_function_name
2	js_object_get_field_by_name
6	singletons

The method that works

Two clusters in, the reliable procedure is:

1. Compile with --debug-symbols, run under PERRY_GC_PROTECT_FROMSPACE=1 + the allocation-point route, take a backtrace.
2. Look for derive-then-collect ordering: a raw pointer or &str taken from a heap object, then a collection point, then a use.
3. If the faulting helper already has a scope or root, the bug is in its caller.

Two things that do not work, both measured:

• Pattern-matching for the shape. A regex for "derive a payload pointer, then allocate within 25 lines" found 34 sites; applying a no-move window to the 28 non-loop-carrying ones closed zero catches (fix(gc): stop reading a string's bytes after the string moved #7363). Only backtrace-found sites were real.
• Fixing the helper when the caller is at fault. See the regex cluster below.

The regex cluster (5) is caller-side, not layer 3

lookup_fancy_regex appears under four different callers, which suggested one shared bug. It is not in that helper:

• Adding a no-move window around its string_as_str + to_string() pair closed 0/5.
• The fault is obj_type=3 (a 32-byte string) at lookup_fancy_regex + 184, which is string_as_str((*re).pattern_ptr).
• RegExpHeader's string slots are correctly traced — regex_gc_slot_ptrs exposes pattern_ptr/flags_ptr as an exact 2-slot range, gated on the header magic (gc/layout.rs).

Those slots are only rewritten if the header itself is visited, i.e. reachable from a root at collection time. And nothing in js_string_replace_regex allocates before the call — frame #2 is generated code. So re was already stale on entry.

Hypothesis for whoever picks this up: a RegExp-typed local is not being classified as pointer-bearing by codegen's root-set analysis, so it never gets a root slot and the header is invisible to the collector. That would make this a layer-2 gap that statepoints inherit rather than fix — the same shape as #7342, where "both root backends faulted identically" because the value had no root slot at all.

Cheap way to test it: compile one of the five with --trace llvm and check whether the regex local gets a shadow/statepoint root.

[proggeramlug]

23 of 31 closed; here is the triage for the remaining 8

Five clusters fixed across four PRs — #7373 (JSON, 8), #7374 (regex flags, 5), #7375 (await, 4), #7376 (symbol description, 6).

Every fix was an ORDERING bug, not a missing root

cluster	root cause
JSON (8)	root pushed after the collection point
regex flags (5)	field stored already stale at construction
await (4)	value re-derived from a pre-collection register
symbol description (6)	field stored already stale at construction

Every one already had rooting of some kind — the pattern's root, the receiver's root, the per-block unbox, the registry for registered symbols. What was missing in each case was ordering the root relative to the collection point. "Add roots" would have been the wrong lesson; "root before the allocation, re-read after" is the right one.

Two of the four were a long-lived object's field written after the allocation that moved it, which is why the most productive rule turned out to be: if a field of a long-lived object is stale, suspect the constructor, not the reader.

Methods that measurably do NOT work

Recorded because each looked obvious and each closed exactly zero:

1. Pattern-matching the shape. A regex for "derive a payload pointer, then allocate within 25 lines" found 34 sites; windowing the 28 non-loop-carrying ones closed 0.
2. Fixing the faulting helper when the constructor is at fault. A no-move window inside lookup_fancy_regex closed 0/5.
3. Rooting the call-site operand for the same cluster. Rooting the search value in codegen's replace lowering closed 0/5, even after confirming in the IR that the root landed and moving the re-read to the point of use.
4. Rooting a parameter when the fault is deeper. Rooting obj across the key allocation in constructor_dynamic_prototype closed 0/3.

Grouping by frame #0 over-counts clusters

js_object_literal_infer_computed_function_name was triaged as its own cluster. It reaches the same stale (*sym_ptr).description through infer_symbol_function_name, so #7376 closed it too. One bad field, two readers, two apparent clusters.

The remaining 8, by stale object type

test	obj_type	frame #0 ← #1
field_lane_semantics	2 OBJECT 72	js_object_get_field_by_name ← ..._f64
put_value_plan_cache	2 OBJECT 72	same
stream_id_band_dynamic_dispatch	2 OBJECT 72	js_object_get_field_by_name ← fetch_subclass_handle_id
diagchannel_3082_…	3 STRING 32	decode_string_value ← throw_tracing_channel_symbol
gc_assign_string_source_rooting	3 STRING 56	mirror_class_object_static_write ← js_object_set_field_by_name
gc_process_env_cache_rooting	3 STRING 40	js_array_push_f64 ← make_iter_result
object_assign_collection	3 STRING 32	js_array_length ← js_object_assign_one
fetch_response	5 PROMISE 72	js_promise_resolve ← js_request_array_buffer

The three OBJECT ones are one group and are harder than what is fixed so far. The fault is at js_object_get_field_by_name + 664 — inside the function, on a 72-byte OBJECT — and rooting the caller's receiver does not close it. That points at the prototype-chain walk holding a link across an allocation, rather than a stale argument. Worth doing next because it is three catches and a hot path; worth budgeting more than the others because the ordering lens alone does not locate it.

object_assign_collection is the case analysed earlier in this issue: js_object_assign_one is already carefully rooted (#7200), so the argument arrives stale — caller-side.

[proggeramlug]

The 3-catch OBJECT group: exact faulting instruction, and two more failed hypotheses

Disassembled it rather than guessing further:

->  0x1005fca38 <+664>: ldurb  w8, [x22, #-0x8]     ; GC header obj_type at user_ptr-8
    0x1005fca3c <+668>: cmp    w8, #0x2             ; == GC_TYPE_OBJECT
    0x1005fca40 <+672>: b.ne   <+1292>

So the fault is the fast lane's GC-header type probe in js_object_get_field_by_name, dereferencing a stale receiver at user_ptr - 8. The fast lane's own comment describes it as resolving a field "with no rooting scope, no exotic-registry probes, and no key hashing" — the absence of rooting is a deliberate design property, which is what makes a stale receiver fatal here rather than merely wrong.

Two hypotheses tested and killed

Both measured 0/3:

1. Stale receiver from constructor_dynamic_prototype. Its js_string_from_bytes(b"constructor") is a real collection point with obj used after it — a textbook instance of the ordering bug fixed four times elsewhere in this issue. Rooting obj across it and re-reading closed nothing.
2. Stale receiver from the process_env_get_field probe at the top of js_object_get_field_by_name, which does an OS lookup and materializes a StringHeader. Rooting obj and key across it and re-reading closed nothing.

Since neither in-function collection point is responsible, obj arrives stale at the call. Frame #1 differs across the three tests — js_object_get_field_by_name_f64 (the typed-this IC-miss fallback) for two of them, fetch_subclass_handle_id for the third — so the next step is the callers, not this function.

Why this group is worth a separate budget

The four clusters already fixed shared one shape: a collection point with a live raw pointer either side of it, locatable by reading ~30 lines around the faulting frame. This one is not that. The receiver is already dead on entry, three different callers reach it, and the consuming code is a fast path that deliberately omits rooting — so the fix is either at each caller or a decision that the fast lane must root after all, which has a measurable cost on the hottest property-read path in the runtime.

Recording the disassembly so whoever picks this up starts from the instruction rather than re-deriving it.