Copying minor loses ObjectHeader::keys_array on large/infrequent cycles — pre-existing, unmasked by #7432; blocks the public baseline

[proggeramlug]

perry's default configuration segfaults on benchmarks/json_polyglot/bench_field_access.ts. It is deterministic (3/3), on main since ~13:00 today, and it blocks regenerating the public benchmark artifact — run_public_baseline.sh completes three of four legs and then fails the cross-runtime checksum gate with "no checksum row for 'perry'", because the default perry (gen-gc + lazy tape) variant produced no successful runs.

Repro

./target/release/perry benchmarks/json_polyglot/bench_field_access.ts -o /tmp/fa
/tmp/fa        # exit 139, no output

Bisect

git bisect over the 46 commits merged today, testing exit status:

first bad commit: 77fa4c5c4  gc: adaptive tenuring + young-scoped scavenge cap (#7432)

Confirmed by hand at both ends — parent 0402a05af exits 0 three times out of three; 77fa4c5c4 and today's tip both exit 139 three out of three.

It is NOT the tenuring threshold

The headline change is ruled out. Pinning tenuring_survivals() to the pre-#7432 constant 4 still crashes 3/3. Flooring the survival-rate lock at 2 instead of 1 also still crashes. So neither adaptive tenuring nor promote-on-first-copy is the mechanism.

What it is: cycle size

The remaining part of #7432 is the young-scoped scavenge cap, which changed the cadence from degenerate once-per-1 MB-block to once per ~16 MB of young-gen occupancy. The effect on this workload:

	copying minors	objects copied per cycle
parent 0402a05af	592	~123k
tip 351742d30	4	228,425 (17.5 MB) in one cycle

PERRY_GC_DIAG on the failing run shows the last two cycles as

copied_objects=228425 copied_bytes=17587656 tenuring_survivals=4
copied_objects=0      copied_bytes=0        tenuring_survivals=1
<segfault>

A cycle copying 228k objects, then one copying zero, then the fault.

Sweeping the cap confirms the size dependence — PERRY_GC_SCAVENGE_NURSERY_MB:

1 MB → 139   2 MB → 139   4 MB → 139   8 MB → 139   16 MB → 139   32 MB → 0

At 32 MB no minor fires in the vulnerable window and the program completes.

The fault

EXC_BAD_ACCESS (code=1, address=0x7ffc000000000014)
->  ldr    w2, [x23, #0x4]
    add    x1, x23, #0x14

0x7FFC_0000_0000_0000 is the NaN-box immediate tag base; +0x14 is the header-payload offset. So a value that should be a heap object is an immediate (undefined/null/bool) and is being dereferenced as a StringHeader/ArrayHeader — length at +4, data at +0x14.

Both preconditions are required

default (gen-gc + lazy tape)   → 139
PERRY_GEN_GC=0                 → 0
PERRY_JSON_TAPE=0              → 0
both off                       → 0

Mark-sweep traces everything, so a missed old→young edge would not bite there — consistent with the copying minor being implicated. LazyArrayHeader holds four raw heap pointers (blob_str, materialized, materialized_elements, materialized_bitmap); they have a GcRewriteDescriptorKind::LazyArray rewrite descriptor in gc/layout.rs, and json_tape.rs registers no scanner of its own. That is where I would look first, at the scale the new cadence produces.

Note on the benchmark's checksum

Unrelated but worth recording: the field_access checksum is not stable run-to-run on the parent build (2552986845, 2552987326, 2552987326), while today's PERRY_JSON_TAPE=0 is stable at 2552986400. A cross-runtime checksum gate over a value that drifts is a weak gate; I did not chase this.

Impact

Default-configuration segfault on a JSON workload, and the public benchmark artifact cannot be regenerated until it is fixed — the gate correctly refuses to publish a comparison in which perry has no row.

[proggeramlug]

Correcting the experimental record above, and adding a build-methodology warning that anyone reproducing this will hit.

Three of my "still crashes" results were run against a stale runtime archive, so they were not evidence of anything. target/release/libperry_runtime.a was a day older than target/release/perry, and — the part that actually bites — the compile linked

target/perry-auto-11d6bccda5436410/release/libperry_runtime.a

i.e. the auto-optimize path builds its own feature-stripped runtime and links that, overriding PERRY_RUNTIME_DIR. A runtime-side patch then looks like a no-op and both arms of an A/B behave identically. Reproducing anything here needs PERRY_NO_AUTO_OPTIMIZE=1, a touch on the edited file, and an mtime check that the .a actually moved.

Re-run with that discipline, one result survives and is now trustworthy:

• Restoring the pre-gc: adaptive tenuring + young-scoped scavenge cap (fixes the large-live-set scavenge regression) #7432 total-arena cap basis does NOT fix it (3/3 crash, fresh archive, auto-opt off).

The two earlier claims I withdraw as untested:

• "pinning tenuring_survivals() to 4 still crashes"
• "flooring the survival-rate lock at 2 still crashes"

Both need re-running under the above discipline before anyone relies on them. I would not assume the tenuring threshold is exonerated on my say-so.

What still stands

Everything in the original report that came from unmodified builds is unaffected, because those were the shipped artifacts rather than my patches:

• bisect to 77fa4c5c4, with parent 0402a05af clean 3/3 and the commit itself 139 3/3
• the cadence/cycle-size contrast (592 small minors → 4 minors, one copying 228,425 objects)
• the PERRY_GC_SCAVENGE_NURSERY_MB sweep (crashes 1–16 MB, passes at 32 MB)
• both preconditions (PERRY_GEN_GC=0 or PERRY_JSON_TAPE=0 each avoid it)
• the fault address decoding to a NaN-box immediate dereferenced as a header

Practical note

A full git revert of #7432 does not apply cleanly — it conflicts in gc/mod.rs because later work (#7443, #7449) built on it. So mitigating by revert is not a one-liner either.

[proggeramlug]

Crash site identified. With --debug-symbols:

EXC_BAD_ACCESS (code=1, address=0x7ffc000000000014)
frame #0: perry_runtime::json::stringify::stringify_object_inner + 3900

So this is JSON stringify, not the tape parse — and 0x7FFC_… is the NaN-box immediate tag base, so a field that must hold a heap pointer holds an immediate.

The unchecked dereference

json/stringify.rs, inside stringify_object_inner:

let keys_arr = (*obj).keys_array;
!keys_arr.is_null() && (*keys_arr).length > num_fields

keys_array is dereferenced behind a null check only. A value that is non-null but not a pointer — which is exactly what a NaN-boxed immediate is — passes that guard and faults. Note the sibling path a few lines below uses object_keys_array_checked(obj), so a validated accessor already exists; this call site does not use it.

Why I think the guard is the symptom, not the bug

ObjectHeader::keys_array is a raw pointer to a GC-managed array. For it to read back as an immediate, something must have written garbage into it or failed to rewrite it. The two shapes worth checking first:

1. A missing old→young remembered-set edge. If the object is promoted to old-gen while its keys_array is still young, the copying minor will not trace that edge unless the write barrier recorded it — the array is then moved or reclaimed without obj.keys_array being rewritten. This fits every observation: PERRY_GEN_GC=0 avoids it (mark-sweep traces everything, so a missing remembered-set entry cannot bite), and gc: adaptive tenuring + young-scoped scavenge cap (fixes the large-live-set scavenge regression) #7432 changes exactly what ends up in old-gen and when.

2. A keys_array slot not covered by the object's rewrite descriptor at the sizes the new cadence produces.

Tightening the call site to object_keys_array_checked would stop the fault but would convert a lost-object bug into silently wrong JSON output, so it should not be the fix on its own.

Corrected sweep, run properly

Re-run with PERRY_NO_AUTO_OPTIMIZE=1 and a verified-fresh archive. The relationship is not monotonic in the cap, so my earlier "passes at 32 MB" was not the whole picture:

PERRY_GC_SCAVENGE_NURSERY_MB	exit	copying minors	max objects/cycle
1	139	6	41,072
4	0	48	66,015
16 (default)	139	4	223,049
32	0	28	65,698
64	139	2	263,317

Crashes at 6/4/2 minors, passes at 48/28. It tracks how few collections happen, not cycle size directly (41k objects crashes, 66k passes) — consistent with "a minor has to land in a particular window", not with a size limit.

Also ruled out, each with a verified-fresh archive

• pinning tenuring_survivals() to the pre-gc: adaptive tenuring + young-scoped scavenge cap (fixes the large-live-set scavenge regression) #7432 constant 4 → still 139 (3/3)
• pinning the influx cap scale to ×1 → still 139 (3/3), still only 4 minors
• restoring the pre-gc: adaptive tenuring + young-scoped scavenge cap (fixes the large-live-set scavenge regression) #7432 total-arena cap basis → still 139 (3/3), cadence only partly restored (12 minors vs the parent's 592)

So none of #7432's three headline levers is individually responsible, which strengthens the "it changed what is in old-gen when, exposing a latent barrier/rewrite gap" reading over "one of its dials is wrong".

[proggeramlug]

This bug is PRE-EXISTING. #7432 exposed it; it did not introduce it. Do not revert #7432.

The bisect was technically correct — 77fa4c5c4 is the first commit where the default configuration crashes — but the defect is older. The parent commit crashes too, as soon as its scavenge cadence is made non-degenerate:

parent 0402a05af	exit	copying minors
PERRY_GC_SCAVENGE_NURSERY_MB=16 (its default)	0	592
PERRY_GC_SCAVENGE_NURSERY_MB=32	139	8

The old cadence was the mask. #7432's own commit message describes it: comparing the cap against arena_total_bytes() meant that once old-gen crossed 16 MB, every fresh 1 MB Eden block re-crossed the trigger, degenerating the cadence to once-per-block. At 592 tiny collections the vulnerable window never opens. #7432 fixed that cadence — correctly — and the latent bug became reachable at default settings.

Old-gen is not involved either

Pinning tenuring_survivals() to 255 so nothing ever promotes (promoted_objects=0, verified via PERRY_GC_DIAG) still crashes 3/3. So my earlier missing-old→young-remembered-set hypothesis is wrong, and I withdraw it. The whole thing happens inside the young generation: the copying minor moves the object and/or its keys_array within Eden/survivor space and ObjectHeader::keys_array comes back holding a NaN-box immediate.

Where that leaves it

The defect is in the copying minor's rewriting of ObjectHeader::keys_array (or in whether the holding object is scanned at all) when a cycle is large and infrequent. It reproduces on any commit — including well before today — given a nursery cap that produces few, large minors.

Two consequences worth stating plainly:

1. Reverting gc: adaptive tenuring + young-scoped scavenge cap (fixes the large-live-set scavenge regression) #7432 would re-mask this, not fix it, and would give back the RSS regression gc: adaptive tenuring + young-scoped scavenge cap (fixes the large-live-set scavenge regression) #7432 was written to fix. The revert also does not apply cleanly (conflicts in gc/mod.rs from gc: old-generation hole free list — swept holes become reusable capacity (#7437) #7443/arena: recycled-block pool — block round-trips through the allocator were the tree.ts RSS term (#7438) #7449).
2. The gc-ratchet probes never caught it because they run at the default cap. A probe that varies PERRY_GC_SCAVENGE_NURSERY_MB — or simply one large-Eden arm — would have. That is the coverage gap this exposes, independent of the fix.

Reproducer, independent of today's merges

git checkout 0402a05af      # or anything older
cargo build --release -p perry -p perry-runtime-static -p perry-stdlib-static
PERRY_RUNTIME_DIR=$PWD/target/release PERRY_NO_AUTO_OPTIMIZE=1 \
  ./target/release/perry benchmarks/json_polyglot/bench_field_access.ts -o /tmp/fa
PERRY_GC_SCAVENGE_NURSERY_MB=32 /tmp/fa    # exit 139