test(gc-ratchet): per-probe gating overrides so the probes run again (#7554) - #7557
Merged
Conversation
added 3 commits
August 7, 2026 03:06
benchmarks/gc_ratchet/tolerances.json is keyed per metric per profile, so the only lever for a metric that has become sample-dependent on ONE probe was to stop gating it on all twelve. Add a probe_overrides section that takes a single (probe, metric) cell out of the gating family, carrying the evidence that justifies the exclusion, and move the bit-identity rule from the unit tests into validate_artifact so such an artifact cannot be pinned in the first place. Refs #7554
Six of the twelve probes pin minor_cycles at 1 and the allowance floor is also 1, so a collapse from 1 to 0 landed on delta == -allowance and scored ok. A collector that stops running copying minors -- the largest regression this ratchet exists to catch -- was reported as passing. Assert liveness instead of inferring it from the bands, and document probe_overrides. Refs #7554
…read The rationale first written for the override named the evacuation policy's RSS and pause thresholds. That was a hypothesis and it is wrong -- no [gc-evac-policy] line is ever emitted on this probe. The measured cause is the conservative stack scan the probe's own explicit gc() forces: diffing two disagreeing traces shows every minor, tenuring decision, step cycle and copy/promote counter matching exactly, with the sole difference in the last mark-sweep's freed_bytes, and PERRY_CONSERVATIVE_STACK_SCAN=off makes the probe bit-identical at 51,668,688 bytes over 8 runs. Refs #7554
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe GC ratchet now supports evidence-backed, probe-specific gating overrides. It validates override structure and artifact consistency, preserves measurement and reporting, rejects inactive collection probes, and documents the ChangesGC ratchet probe gating overrides
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Tolerances
participant GC_Ratchet
participant Artifact
participant Report
Tolerances->>GC_Ratchet: Load probe_overrides
GC_Ratchet->>Artifact: Validate references and deterministic metrics
GC_Ratchet->>GC_Ratchet: Resolve per-probe tolerances
GC_Ratchet->>GC_Ratchet: Fail inactive collection probes
GC_Ratchet->>Report: Render excluded cells and evidence
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
tests/test_gc_ratchet.py (2)
464-464: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winIterate
RETENTION_METRICSinstead of a hardcoded tuple.The test hardcodes
("heap_used_bytes", "heap_total_bytes").validate_artifactenforces the same rule overDETERMINISTIC_METRICS, which isRETENTION_METRICS + GC_METRICS. If a retention metric is added later, this test keeps its old coverage without any signal. Import and iterateRETENTION_METRICSto keep the test aligned with the constant it mirrors.♻️ Proposed refactor
- for metric in ("heap_used_bytes", "heap_total_bytes"): + for metric in RETENTION_METRICS:Add the import alongside the existing ones:
RETENTION_METRICS,🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_gc_ratchet.py` at line 464, Replace the hardcoded metric tuple in the test loop with the existing RETENTION_METRICS constant, and add RETENTION_METRICS to the relevant imports. Keep the loop’s validation behavior unchanged so it automatically covers all retention metrics.
574-598: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider covering the remaining rejection branches.
_probe_override_from_jsonalso rejects unknown entry fields, unknownevidencefields, missing individual evidence fields, and blankmeasured_onorissue. No test exercises those branches. A future edit could drop one of those checks without any test turning red. Add one parametrized test over the malformed variants.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_gc_ratchet.py` around lines 574 - 598, The override validation tests around _probe_override_from_json are missing coverage for malformed metadata. Add one parametrized test covering unknown entry fields, unknown evidence fields, each missing required evidence field, and blank measured_on or issue values; assert that each variant raises RatchetError, while reusing the existing _override_entry and _with_override helpers.benchmarks/gc_ratchet/gc_ratchet.py (1)
1024-1029: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePrint the observed value instead of a hardcoded
0.The condition accepts any
median <= 0, but the message always prints-> 0. If a negative median ever reaches this branch, the failure text states a value that was not measured.♻️ Proposed refactor
failures.append( f"{name}: {what} in this run ({metric} " - f"{base_entry['metrics'][metric]['median']:,.0f} -> 0). The baseline it is " + f"{base_entry['metrics'][metric]['median']:,.0f} -> " + f"{cur_entry['metrics'][metric]['median']:,.0f}). The baseline it is " "being compared against measures a collector that did; there is nothing " "here to compare." )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/gc_ratchet/gc_ratchet.py` around lines 1024 - 1029, Update the failure message in the median comparison branch to interpolate the actual observed median value instead of the hardcoded 0. Preserve the existing baseline metric formatting and explanatory text, using the same median value that satisfied the <= 0 condition.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/gc_ratchet/gc_ratchet.py`:
- Around line 669-687: Update the evidence validation function around the
observed_runs and observed_spread conversions to catch non-numeric or null
values and re-raise them as RatchetError using the existing where prefix.
Preserve the current minimum-run and positive-spread checks after successful
conversion, and ensure both int() and float() conversion failures are handled
consistently with the other schema violations.
- Around line 1109-1113: Update the report formatting in the override line
construction to preserve non-zero fractional observed_spread values instead of
formatting them with zero decimal places. Keep the existing context and
thousands separators while ensuring values such as 0.4 remain visibly greater
than zero.
In `@benchmarks/gc_ratchet/README.md`:
- Around line 193-207: Update the retention-table documentation to account for
the per-probe exception on 12_large_live_set.heap_used_bytes: identify its
shipped baseline spread of 6768 and note that its probe override sets gating to
false, rather than describing heap_used_bytes as universally gated with 0.000%
spread. Keep the general profile-level table accurate for other probes.
In `@benchmarks/gc_ratchet/tolerances.json`:
- Around line 211-214: Align the duplicated override evidence by correcting the
observed_spread in benchmarks/gc_ratchet/tolerances.json at lines 211-214, using
9744 if all documented runs contribute or documenting the narrower population
consistently. Apply the identical correction to
benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json at lines 282-285.
---
Nitpick comments:
In `@benchmarks/gc_ratchet/gc_ratchet.py`:
- Around line 1024-1029: Update the failure message in the median comparison
branch to interpolate the actual observed median value instead of the hardcoded
0. Preserve the existing baseline metric formatting and explanatory text, using
the same median value that satisfied the <= 0 condition.
In `@tests/test_gc_ratchet.py`:
- Line 464: Replace the hardcoded metric tuple in the test loop with the
existing RETENTION_METRICS constant, and add RETENTION_METRICS to the relevant
imports. Keep the loop’s validation behavior unchanged so it automatically
covers all retention metrics.
- Around line 574-598: The override validation tests around
_probe_override_from_json are missing coverage for malformed metadata. Add one
parametrized test covering unknown entry fields, unknown evidence fields, each
missing required evidence field, and blank measured_on or issue values; assert
that each variant raises RatchetError, while reusing the existing
_override_entry and _with_override helpers.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 27267074-d096-48d6-8cc0-93182819dfda
📒 Files selected for processing (5)
benchmarks/gc_ratchet/README.mdbenchmarks/gc_ratchet/baseline/gc-ratchet-v1.jsonbenchmarks/gc_ratchet/gc_ratchet.pybenchmarks/gc_ratchet/tolerances.jsontests/test_gc_ratchet.py
Comment on lines
+669
to
+687
| for field in ("observed_runs", "observed_spread", "measured_on", "issue"): | ||
| if field not in evidence: | ||
| raise RatchetError(f"{where}: evidence is missing {field}") | ||
| runs = int(evidence["observed_runs"]) | ||
| if runs < MIN_EXCLUSION_RUNS: | ||
| raise RatchetError( | ||
| f"{where}: evidence rests on {runs} runs; at least {MIN_EXCLUSION_RUNS} are " | ||
| "required, the same number every band in this file is justified by. Fewer runs " | ||
| "cannot distinguish a non-deterministic metric from one bad sample." | ||
| ) | ||
| spread = float(evidence["observed_spread"]) | ||
| if spread <= 0: | ||
| raise RatchetError( | ||
| f"{where}: evidence records a spread of {spread:g}. A metric that was observed " | ||
| "to be deterministic has not been shown to be ungateable; it must stay gated." | ||
| ) | ||
| for field in ("measured_on", "issue"): | ||
| if not str(evidence[field]).strip(): | ||
| raise RatchetError(f"{where}: evidence.{field} is blank") |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Convert non-numeric evidence values into RatchetError.
int(evidence["observed_runs"]) and float(evidence["observed_spread"]) raise ValueError or TypeError for a string or null in tolerances.json. Every other schema violation in this function raises RatchetError with the probe_overrides.<probe>.<metric> prefix. A mistyped evidence value therefore escapes the CLI's RatchetError handling and surfaces as a traceback without the cell name.
🛠️ Proposed fix
- runs = int(evidence["observed_runs"])
+ try:
+ runs = int(evidence["observed_runs"])
+ except (TypeError, ValueError):
+ raise RatchetError(
+ f"{where}: evidence.observed_runs is not a number "
+ f"({evidence['observed_runs']!r})"
+ ) from None
if runs < MIN_EXCLUSION_RUNS:
@@
- spread = float(evidence["observed_spread"])
+ try:
+ spread = float(evidence["observed_spread"])
+ except (TypeError, ValueError):
+ raise RatchetError(
+ f"{where}: evidence.observed_spread is not a number "
+ f"({evidence['observed_spread']!r})"
+ ) from None📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| for field in ("observed_runs", "observed_spread", "measured_on", "issue"): | |
| if field not in evidence: | |
| raise RatchetError(f"{where}: evidence is missing {field}") | |
| runs = int(evidence["observed_runs"]) | |
| if runs < MIN_EXCLUSION_RUNS: | |
| raise RatchetError( | |
| f"{where}: evidence rests on {runs} runs; at least {MIN_EXCLUSION_RUNS} are " | |
| "required, the same number every band in this file is justified by. Fewer runs " | |
| "cannot distinguish a non-deterministic metric from one bad sample." | |
| ) | |
| spread = float(evidence["observed_spread"]) | |
| if spread <= 0: | |
| raise RatchetError( | |
| f"{where}: evidence records a spread of {spread:g}. A metric that was observed " | |
| "to be deterministic has not been shown to be ungateable; it must stay gated." | |
| ) | |
| for field in ("measured_on", "issue"): | |
| if not str(evidence[field]).strip(): | |
| raise RatchetError(f"{where}: evidence.{field} is blank") | |
| for field in ("observed_runs", "observed_spread", "measured_on", "issue"): | |
| if field not in evidence: | |
| raise RatchetError(f"{where}: evidence is missing {field}") | |
| try: | |
| runs = int(evidence["observed_runs"]) | |
| except (TypeError, ValueError): | |
| raise RatchetError( | |
| f"{where}: evidence.observed_runs is not a number " | |
| f"({evidence['observed_runs']!r})" | |
| ) from None | |
| if runs < MIN_EXCLUSION_RUNS: | |
| raise RatchetError( | |
| f"{where}: evidence rests on {runs} runs; at least {MIN_EXCLUSION_RUNS} are " | |
| "required, the same number every band in this file is justified by. Fewer runs " | |
| "cannot distinguish a non-deterministic metric from one bad sample." | |
| ) | |
| try: | |
| spread = float(evidence["observed_spread"]) | |
| except (TypeError, ValueError): | |
| raise RatchetError( | |
| f"{where}: evidence.observed_spread is not a number " | |
| f"({evidence['observed_spread']!r})" | |
| ) from None | |
| if spread <= 0: | |
| raise RatchetError( | |
| f"{where}: evidence records a spread of {spread:g}. A metric that was observed " | |
| "to be deterministic has not been shown to be ungateable; it must stay gated." | |
| ) | |
| for field in ("measured_on", "issue"): | |
| if not str(evidence[field]).strip(): | |
| raise RatchetError(f"{where}: evidence.{field} is blank") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/gc_ratchet/gc_ratchet.py` around lines 669 - 687, Update the
evidence validation function around the observed_runs and observed_spread
conversions to catch non-numeric or null values and re-raise them as
RatchetError using the existing where prefix. Preserve the current minimum-run
and positive-spread checks after successful conversion, and ensure both int()
and float() conversion failures are handled consistently with the other schema
violations.
Comment on lines
+1109
to
+1113
| lines.append( | ||
| f"- `{probe}`.{metric} — {override.rationale} " | ||
| f"(observed spread {override.observed_spread:,.0f} over " | ||
| f"{override.observed_runs} runs, {override.measured_on}; {override.issue})" | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not round observed_spread to zero in the report.
observed_spread is a float, and the schema only requires it to be greater than zero. An override on a fractional metric, for example wall_ms with a spread of 0.4, renders as "observed spread 0 over 21 runs". That text contradicts the non-zero-spread rule the reader is being shown, and it hides the evidence that justifies the exclusion.
🛠️ Proposed fix
lines.append(
f"- `{probe}`.{metric} — {override.rationale} "
- f"(observed spread {override.observed_spread:,.0f} over "
+ f"(observed spread {override.observed_spread:,.6g} over "
f"{override.observed_runs} runs, {override.measured_on}; {override.issue})"
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| lines.append( | |
| f"- `{probe}`.{metric} — {override.rationale} " | |
| f"(observed spread {override.observed_spread:,.0f} over " | |
| f"{override.observed_runs} runs, {override.measured_on}; {override.issue})" | |
| ) | |
| lines.append( | |
| f"- `{probe}`.{metric} — {override.rationale} " | |
| f"(observed spread {override.observed_spread:,.6g} over " | |
| f"{override.observed_runs} runs, {override.measured_on}; {override.issue})" | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/gc_ratchet/gc_ratchet.py` around lines 1109 - 1113, Update the
report formatting in the override line construction to preserve non-zero
fractional observed_spread values instead of formatting them with zero decimal
places. Keep the existing context and thousands separators while ensuring values
such as 0.4 remain visibly greater than zero.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@changelog.d/7557-gc-ratchet-per-probe-gating.md`:
- Around line 53-57: Update the changelog statement around the probe results to
remove the claim that retention reproduced bit-for-bit across hosts. State that
the GC counters reproduced bit-for-bit, while retention for
12_large_live_set.heap_used_bytes remained sample-dependent under the default
conservative stack scan.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 0ef6d042-c68d-4d67-93fa-26274f0d8969
📒 Files selected for processing (3)
benchmarks/gc_ratchet/baseline/gc-ratchet-v1.jsonbenchmarks/gc_ratchet/tolerances.jsonchangelog.d/7557-gc-ratchet-per-probe-gating.md
🚧 Files skipped from review as they are similar to previous changes (2)
- benchmarks/gc_ratchet/baseline/gc-ratchet-v1.json
- benchmarks/gc_ratchet/tolerances.json
Comment on lines
+53
to
+57
| - **The probes run again.** Full `measure` + `check`, twelve probes each, on two | ||
| machine classes: the pinned quiet host (`perry-macos`, M1 mini, load 2.2, | ||
| `pinned_host` profile) and a MacBook Pro under load 21 (`shared_ci`). All | ||
| twenty-four probe runs compiled, ran, and passed their Node-oracle diff, and | ||
| retention and the GC counters reproduced bit-for-bit across the two hosts. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Correct the bit-identity claim.
Line 57 says that retention reproduced bit-for-bit across both hosts. Lines 70-80 document non-zero retention variance for 12_large_live_set.heap_used_bytes under the default conservative stack scan. State that the GC counters reproduced bit-for-bit, while this retention metric remained sample-dependent.
Proposed wording
- GC counters reproduced bit-for-bit across the two hosts.
+ GC counters reproduced bit-for-bit across the two hosts, while
+ `12_large_live_set.heap_used_bytes` remained sample-dependent under the
+ default conservative stack scan.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - **The probes run again.** Full `measure` + `check`, twelve probes each, on two | |
| machine classes: the pinned quiet host (`perry-macos`, M1 mini, load 2.2, | |
| `pinned_host` profile) and a MacBook Pro under load 21 (`shared_ci`). All | |
| twenty-four probe runs compiled, ran, and passed their Node-oracle diff, and | |
| retention and the GC counters reproduced bit-for-bit across the two hosts. | |
| - **The probes run again.** Full `measure` + `check`, twelve probes each, on two | |
| machine classes: the pinned quiet host (`perry-macos`, M1 mini, load 2.2, | |
| `pinned_host` profile) and a MacBook Pro under load 21 (`shared_ci`). All | |
| twenty-four probe runs compiled, ran, and passed their Node-oracle diff, and | |
| retention and the GC counters reproduced bit-for-bit across the two hosts, | |
| while `12_large_live_set.heap_used_bytes` remained sample-dependent under the | |
| default conservative stack scan. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@changelog.d/7557-gc-ratchet-per-probe-gating.md` around lines 53 - 57, Update
the changelog statement around the probe results to remove the claim that
retention reproduced bit-for-bit across hosts. State that the GC counters
reproduced bit-for-bit, while retention for 12_large_live_set.heap_used_bytes
remained sample-dependent under the default conservative stack scan.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the blocker in #7554:
gc-ratchethas measured nothing since 2026-08-05because
test_pinned_artifact_retention_is_deterministicfails in the"Harness unit tests and artifact validation" step, which runs before the
measurement step. All twelve probes have been skipped on every branch.
The assertion is correct and is not weakened here.
12_large_live_setretention really is sample-dependent, and its band is justified in
tolerances.jsonas anti-brittleness margin over an observed spread of0.000%, not as a noise allowance. The bug is that the artifact was pinned as
if the cell could carry a gate.
The lever the assertion asks for did not exist
tolerances.jsonis keyed per metric per profile, so "take12_large_live_set.heap_used_bytesout of the gating family" — what theassertion's own message instructs — could only be said by turning
heap_used_bytesgating off for all twelve probes, discarding a bit-identicalsignal on the other eleven.
probe_overridesA band expresses a machine class's noise floor, which is per profile. Whether a
metric is deterministic enough to gate at all is a property of the workload,
which is per probe. Those two were conflated; they are now separate.
Every property is a refusal, because an exclusion is a hole in a gate:
gatingmay only be set tofalse. Re-gating stays the profile's job,where a reader looking for what is gated will find it.
compared, and still printed — a breach shows as
drift (informational)rather than vanishing.
checkprints every override with its full reasonunder the table on every run, so a
noin the Gating column can be explainedwithout opening another file.
every band in the file is justified by, and a spread that is actually
non-zero. You cannot exclude a metric you have not shown is ungateable.
scripts/gc_root_dominance_allowlist.jsonalready carries. Fixing thenon-determinism means deleting the entry.
that is a profile-level
"gating": falsewith nowhere to read the reason.The bit-identity rule also moves from the unit tests into
validate_artifact,so an artifact carrying a non-deterministic gating cell can no longer be
pinned. Had that rule been there on 2026-08-05, the re-pin would have failed
loudly on the maintainer's machine instead of silently wedging CI for two days.
Sabotage-checked: deleting the override from the artifact makes
validaterefuse it, so the rule is not vacuous.
A second gate hole, found on the way
checknow fails a probe whose current run reportsminor_cycles == 0orcopied_objects == 0where the baseline reports more. The tolerancearithmetic could not catch this: six of the twelve probes pin
minor_cyclesat 1 and the allowance floor is also 1, so a collapse from 1 to 0 is
delta == -allowanceand scoredok. A collector that stopped runningcopying minors — the largest regression this ratchet exists to catch — was
being reported as passing. That is CLAUDE.md's fourth failure mode sitting
inside the gate built to close it.
The probes run again, and the gate is doing its job
End-to-end, twice, with
PERRY_NO_AUTO_OPTIMIZE=1and the pinned Node oracle(v26.5.1); all twelve probes compiled, ran, and passed their oracle diff in
both:
perry-macos, M1 mini, load 2.2)pinned_hostshared_cipasspassRetention and the counters reproduced bit-for-bit across the two machine
classes, exactly as the README claims they do. The ten breaches are real and
have been invisible since 2026-08-05:
Most of these read as improvements wearing a two-sided band:
03and04shed40–95% of their copy/promote work while their retention fell 49% and 22%, so
objects that used to be copied and tenured are now recognised as dead. Two are
not:
05_closure_captureretains +16.44% more with every collector counterunchanged (
copied_objects,copied_bytes,promoted_*andfreed_bytesall
+0.00%) — same work, more retained — and02_survivor_promotionis+2.77% on the same shape.
This PR deliberately does not re-pin. Re-pinning to turn a red gate green
is what the artifact exists to prevent, and
05_closure_captureis a retentionincrease that should be looked at before it is accepted. The gate now produces
the table that makes that decision possible.
The 12_large_live_set non-determinism, diagnosed
Not benign, and not what it looked like. Every probe reads
process.memoryUsage()after an explicitgc()— and an explicitgc()runs afull mark-sweep with a forced conservative stack scan
(
[gc-scan-fallback] site=manual_collect automatic=false, printed on everyrun). A conservative scan retains whatever the native stack looks like a
pointer to, and stack residue differs run to run.
Diffing two
PERRY_GC_DIAGtraces that disagree shows it precisely: theminors, the tenuring decisions, the step cycles and every copy/promote counter
match exactly, and the sole difference is the last collection's
freed_bytes. The clincher:So the variance is entirely false roots — and the conservative scan is
systematically retaining 8.28 MB, 16% of this probe's reported retention.
The eleven small probes stay bit-identical because their live sets are one to
two orders of magnitude smaller, so a stale stack word is much less likely to
alias a plausible heap address at all. Worth its own issue: the retention
metric is measured after a deliberately conservative collection, so on a large
heap it reports precise retention plus a stack-residue tax.
Reproduced three times, independently: pinned host spread 2,304 over 7 harness
repeats (all eleven other probes spread 0), MacBook spread 9,072 over 7, and 22
ad-hoc runs spanning 9,744 bytes.
The ~29% retention win does not reproduce
Checked carefully, because it would have been worth recording. It does not
hold, and one premise of the observation was already off: the pinned artifact
was captured on
perry-macos.fritz.box— the same Mac mini as the 21-runexperiment, not a separate bench host (
notesin the artifact call it "thededicated bench host … which replaces the shared MacBook"). So there was no
host difference to control for.
Three measurements of
12_large_live_set.heap_used_bytes, all with theharness's own protocol:
5e236e6e2, 2026-08-05perry-macos52f7dae1f(the commit the 42.6 MB was read at), releasebe1fc80f8(main + this PR)perry-macosRetention on that probe has not moved at all across the whole 2026-08-06 batch.
PERRY_GC_DIAG=1does not change it (the harness's traced/untraced split stillholds), and neither does the auto-optimizer. There is no 29% win to record;
whatever produced 42.6 MB was not this probe under the harness protocol at that
commit.
Validation
Local only, and stated plainly: CI has a deep runner backlog and may not report.
python3 -m unittest tests.test_gc_ratchet— 54 tests, OK (was 47; the newones are mostly refusals)
python3 benchmarks/gc_ratchet/gc_ratchet.py validate— OK, and refuses thesame artifact with the override removed
measure+checkend to end on two hosts, twelve probes each,as above
python3 scripts/raw_handle_debt.py— 998 (baseline 998)cargo fmt --all -- --check— cleanscripts/check_file_size.sh— cleanNo version bump (maintainer bumps at merge).
Summary by CodeRabbit