[Perf] Linux/arm64: 6 Regressions on 7/17/2026 11:45:18 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	929271670357bdecd4454c3d126c6b0afc581aa5
Compare	3522269feadf7e6d9a5ed422ccd269335ebe62a9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
SingleParallel - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	55.13 ns	86.21 ns	1.56	0.14	False
MultipleSerial - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	318.01 ns	354.46 ns	1.11	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests<Object>*'

Details

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True)

ETL Files

Histogram

JIT Disasms

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	929271670357bdecd4454c3d126c6b0afc581aa5
Compare	3522269feadf7e6d9a5ed422ccd269335ebe62a9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<Int16>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ToFrozenDictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	23.83 μs	28.10 μs	1.18	0.00	False
ToFrozenDictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	2.61 μs	3.02 μs	1.16	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen<Int16>*'

Details

System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary(Count: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary(Count: 64)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	929271670357bdecd4454c3d126c6b0afc581aa5
Compare	3522269feadf7e6d9a5ed422ccd269335ebe62a9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<NotKnownComparable>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ToFrozenDictionary - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	26.93 μs	32.16 μs	1.19	0.00	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen<NotKnownComparable>*'

Details

System.Collections.Perf_Frozen<NotKnownComparable>.ToFrozenDictionary(Count: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 22.04
Queue	AmpereUbuntu
Baseline	929271670357bdecd4454c3d126c6b0afc581aa5
Compare	3522269feadf7e6d9a5ed422ccd269335ebe62a9
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.BinaryWriterExtendedTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
WriteAsciiCharArray - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	26.38 ns	33.15 ns	1.26	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*'

Details

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 32)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True) — caused by commit 2119b86 ("Add NoInlining to ArrayPool Rent/Return (#129319)"). 5 tests bisected to related culprit commits. ⚠️ Note: One or more tests show bimodal history, but at least one clean test bisected to a related culprit.

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 2119b868df7afa9c480058af4912188914e64ffe

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 3 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (overconfident-change, bimodal-history)

📊 Full Analysis (click to expand)

Summary: 4 of 6 reported tests are confirmed regressions on Linux/arm64: three System.Collections.Perf_Frozen.ToFrozenDictionary variants (+16% to +19%, z=16.8–20.7, ultra-stable history) and RentReturnArrayPoolTests<Object>.MultipleSerial (+11.5%, z=2.85). The ArrayPool regression is attributed with high confidence to 2119b868 ("Add NoInlining to ArrayPool Rent/Return"), which surgically adds call overhead to ConfigurableArrayPool. The Frozen dictionary regressions are arm64-only step changes most plausibly caused by an arm64 JIT codegen change (d61e4857); attribution is moderate and bisection is recommended. The remaining two tests (SingleParallel, WriteAsciiCharArray) are measurement noise.

Issue Overview

• Issue: [Perf] Linux/arm64: 6 Regressions on 7/17/2026 11:45:18 PM +00:00 #131277 - [Perf] Linux/arm64: 6 Regressions on 7/17/2026 11:45:18 PM +00:00
• Date: 2026-07-17 | Queue: AmpereUbuntu (Ubuntu.2204.Arm64.Perf) | OS/Arch: ubuntu 22.04/arm64
• Commits: 9292716 → 3522269 (49 commits)
• Labels: untriaged, perf-regression, arch-arm64, os-linux, runtime-coreclr, ampere, kind-micro, compilationmode-tiered, runkind-micro
• Runtime stack: Desktop + CoreCLR (arm64); regression is arm64-only.
• Historical Data: Available (298–300 prior runs per test; 1262 runs examined total)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
RentReturnArrayPoolTests<Object>.SingleParallel (4096, shared)	55.13 ns	86.21 ns	+56.4%	73.62	22.09	300	0.57	60.0%	Y	Bimodal history (two regimes); compare falls between regimes
RentReturnArrayPoolTests<Object>.MultipleSerial (4096, non-shared)	318.01 ns	354.46 ns	+11.5%	322.11	11.37	300	2.85	7.1%	N	Step change, moderate stability (CV=3.5%)
Perf_Frozen<Int16>.ToFrozenDictionary (512)	23.83 μs	28.10 μs	+17.9%	24018	243	298	16.82	5%	N	Step change, ultra-stable (CV=1.0%)
Perf_Frozen<Int16>.ToFrozenDictionary (64)	2.61 μs	3.02 μs	+16.0%	2630	23	300	16.93	5%	N	Step change, ultra-stable (CV=0.9%)
Perf_Frozen<NotKnownComparable>.ToFrozenDictionary (512)	26.93 μs	32.16 μs	+19.4%	26895	254	300	20.72	5%	N	Step change, ultra-stable (CV=0.9%)
BinaryWriterExtendedTests.WriteAsciiCharArray (32)	26.38 ns	33.15 ns	+25.7%	29.58	5.74	300	0.62	38.8%	Y	High-variance history (CV=19%); partial reversion to mean

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Strong historical data (298–300 runs/test), low variance on the confirmed Frozen regressions, complete commit range. Not verified via rerun/bisection, so not 5.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
2119b868	src/libraries/System.Private.CoreLib/.../Buffers/ConfigurableArrayPool.cs	Relevant	Adds [MethodImpl(NoInlining)] to Rent/Return — directly on the ArrayPool hot path (BCL, all stacks).
d61e4857	src/coreclr/jit/codegenarmarch.cpp, emitarm.cpp, lsraarm.cpp, assertionprop.cpp, gentree.cpp, importercalls.cpp	Relevant	arm64-specific JIT codegen + LSRA/assertionprop changes; can alter general integer codegen quality on arm64 (matches arm64-only regression).
475838a7	src/coreclr/jit/gentree.cpp	Relevant	Stricter GenTreeCall::Equals (compares more call flags) can reduce call CSE/dedup → more redundant calls; cross-platform, plausible for call-heavy construction paths.
5a5020c6	src/coreclr/jit/fgbasic.cpp, importercalls.cpp, namedintrinsiclist.h	Relevant	New RuntimeHelpers.IsRuntimeAsync intrinsic; JIT importer change, lower probability.

Confirmed Regressions

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.MultipleSerial (RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False) (+11.5%)

• Delta: 318.01 ns → 354.46 ns (+11.46%)
• Historical Context: μ=322.11 (σ=11.37, n=300); z-score=2.85; adaptive threshold=7.06%
• Assessment: Step change, moderate stability (CV=3.5%)
• Finding Confidence: 4/5 - 300 runs, clear separation of compare from before mean.
• Regression Confidence: 5/5 - Surgical, directly-relevant code change; test uses the non-shared pool (UseSharedPool: False = ConfigurableArrayPool), exactly the type modified.
• Likely Cause: 2119b868 - "Add NoInlining to ArrayPool Rent/Return (Add NoInlining to ArrayPool Rent/Return #129319)"
  • Affected Areas: Collections / Buffers (BCL ArrayPool)
  • Stack Relevance: Relevant (BCL, exercised by CoreCLR arm64)
  • Rationale: The commit adds [MethodImpl(MethodImplOptions.NoInlining)] to ConfigurableArrayPool.Rent/Return. MultipleSerial performs many serial Rent/Return calls on the non-shared configurable pool, so forcing out-of-line calls adds per-call overhead — directly consistent with a ~+11% step change. Authored/committed 2026-07-17, aligning with the regression date. SingleParallel (shared pool) is dominated by contention and shows only noise here, consistent with the change primarily affecting the serial per-call path.
  • Candidates Considered:
    • 2119b868 - Surgical NoInlining on the exact pool type used; best fit.
    • d61e4857 - arm64 JIT codegen; possible but non-specific to ArrayPool.
• Verification Status: Not verified (rerun/bisection recommended to confirm)

System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary (Count: 512) (+17.9%)

• Delta: 23.83 μs → 28.10 μs (+17.91%)
• Historical Context: μ=24018 ns (σ=243, n=298); z-score=16.82; adaptive threshold=5%
• Assessment: Step change, ultra-stable history (CV=1.0%); all 16 after-values outside the entire before range
• Finding Confidence: 4/5 - Very tight history, unambiguous step.
• Regression Confidence: 3/5 - Signal is unambiguous, but attribution is uncertain: 49-commit range with several plausible JIT candidates and no direct code hit for FrozenDictionary. Capped at 3 pending verification.
• Likely Cause: d61e4857 - "JIT, AOT preinit, and CoreLib: Saturate float/double conversions to small integral types (JIT, AOT preinit, and CoreLib: Saturate float/double conversions to small integral types #128604)"
  • Affected Areas: JIT (arm64 codegen, LSRA, assertion propagation)
  • Stack Relevance: Relevant (CoreCLR arm64 JIT)
  • Rationale: The regression is arm64-only and a clean step change, pointing to a codegen-quality change rather than a library edit (no commit touches System.Collections.Frozen). This commit is the only one modifying arm64-specific codegen (codegenarmarch.cpp, emitarm.cpp, lsraarm.cpp) alongside broad JIT files (gentree.cpp, assertionprop.cpp, importercalls.cpp); such changes can perturb register allocation / assertion propagation for the integer-heavy FrozenDictionary construction loops. Attribution is not verified — 475838a7 (call CSE/dedup) is a viable cross-platform alternative.
  • Candidates Considered:
    • d61e4857 - Only arm64-specific codegen change in range; matches arm64-only regression.
    • 475838a7 - Stricter GenTreeCall::Equals may reduce call dedup on call-heavy construction paths.
    • 5a5020c6 - Importer change for new async intrinsic; lower probability.
• Verification Status: Not verified — bisection strongly recommended (see Recommended Actions)

System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary (Count: 64) (+16.0%)

• Delta: 2.61 μs → 3.02 μs (+15.98%)
• Historical Context: μ=2630 ns (σ=23, n=300); z-score=16.93; adaptive threshold=5%
• Assessment: Step change, ultra-stable history (CV=0.9%); all 16 after-values outside before range
• Finding Confidence: 4/5 - Very tight history.
• Regression Confidence: 3/5 - Same shared-code-path signal as the 512 variant; attribution uncertain, capped pending verification.
• Likely Cause: d61e4857 - "JIT, AOT preinit, and CoreLib: Saturate float/double conversions to small integral types (JIT, AOT preinit, and CoreLib: Saturate float/double conversions to small integral types #128604)"
  • Affected Areas: JIT (arm64 codegen)
  • Stack Relevance: Relevant (CoreCLR arm64 JIT)
  • Rationale: Same ToFrozenDictionary<Int16> construction path as the 512-count variant; both regress by the same magnitude and date, corroborating a single JIT codegen cause on arm64. See the 512 variant for the full candidate discussion.
  • Candidates Considered:
    • d61e4857 - arm64 codegen.
    • 475838a7 - call dedup change.
• Verification Status: Not verified — bisection recommended

System.Collections.Perf_Frozen<NotKnownComparable>.ToFrozenDictionary (Count: 512) (+19.4%)

• Delta: 26.93 μs → 32.16 μs (+19.41%)
• Historical Context: μ=26895 ns (σ=254, n=300); z-score=20.72; adaptive threshold=5%
• Assessment: Step change, ultra-stable history (CV=0.9%); after-values (30952–32182) non-overlapping with before range (26418–27810)
• Finding Confidence: 4/5 - Very tight history; highest z-score in the set.
• Regression Confidence: 3/5 - Strongest signal, but same attribution uncertainty as the other Frozen tests; capped pending verification.
• Likely Cause: d61e4857 - "JIT, AOT preinit, and CoreLib: Saturate float/double conversions to small integral types (JIT, AOT preinit, and CoreLib: Saturate float/double conversions to small integral types #128604)"
  • Affected Areas: JIT (arm64 codegen)
  • Stack Relevance: Relevant (CoreCLR arm64 JIT)
  • Rationale: Third ToFrozenDictionary variant regressing together on the same date and platform strengthens the case for a shared JIT codegen cause on arm64. NotKnownComparable uses a comparer-based (non-default-comparable) construction path yet regresses similarly, consistent with a general codegen change rather than a type-specific library edit.
  • Candidates Considered:
    • d61e4857 - arm64 codegen.
    • 475838a7 - call dedup change.
• Verification Status: Not verified — bisection recommended

Open Questions

• Historical Data: Available and strong (298–300 runs/test); no data gaps.
• Confidence Limitations: Frozen dictionary attribution is uncertain — 49-commit range, multiple plausible JIT candidates, and no commit directly touches System.Collections.Frozen. ArrayPool attribution is high-confidence (direct code hit).
• Verification Needed: Bisection of the arm64 JIT candidates for the three Frozen regressions; isolated rerun of the ArrayPool MultipleSerial change to confirm the NoInlining overhead.
• Bisection Status: Not performed (deferred; will be handled separately).

Recommended Actions

1. Bisection (Frozen) - Run bisection on commits 9292716...3522269 using RunWithBuildAtHash for System.Collections.Perf_Frozen<Int16>.ToFrozenDictionary(Count: 512) on arm64 to isolate the JIT culprit among d61e4857 / 475838a7.
2. Verification (ArrayPool) - Verify the regression with RunWithBuildAtHash at 2119b868 vs its parent for RentReturnArrayPoolTests<Object>.MultipleSerial to confirm the NoInlining overhead.
3. Follow-up - Consider collecting hardware counters (collectHardwareCounters=true) during bisection to distinguish added call overhead vs register-allocation/codegen regressions in the Frozen construction loops.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
SingleParallel(RentalSize: 4096, Mani...	✅ Culprit found	2119b868df7a	✅ RELATED	55.1→86.2 ns (+56%)	53.0→88.5 ns (+67%)	1420s
MultipleSerial(RentalSize: 4096, Mani...	❌ Not measured (timeout)	-	-	318.0→354.5 ns (+11%)	— (not measured)	9000s
ToFrozenDictionary(Count: 512)	✅ Culprit found	2119b868df7a	✅ RELATED	23.8→28.1 μs (+18%)	23.7→28.5 μs (+20%)	4046s
ToFrozenDictionary(Count: 64)	✅ Culprit found	2119b868df7a	✅ RELATED	2.6→3.0 μs (+16%)	2.6→3.0 μs (+16%)	1174s
ToFrozenDictionary(Count: 512)	✅ Culprit found	2119b868df7a	✅ RELATED	26.9→32.2 μs (+19%)	26.9→31.4 μs (+17%)	1821s
WriteAsciiCharArray(StringLengthInCha...	✅ Culprit found	2119b868df7a	✅ RELATED	26.4→33.1 ns (+26%)	26.4→33.2 ns (+26%)	1407s

Total bisection time: 18869s
Success rate: 5/6 tests

System.Buffers.Tests.RentReturnArrayPoolTests.SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True)

• Culprit: 2119b868df7a
• Commit Message: Add NoInlining to ArrayPool Rent/Return (Add NoInlining to ArrayPool Rent/Return #129319)
• Relevance: RELATED
• Explanation: The commit adds [MethodImpl(MethodImplOptions.NoInlining)] to Rent and Return in SharedArrayPool.cs (and ConfigurableArrayPool.cs). The regression test RentReturnArrayPoolTests.SingleParallel with UseSharedPool: True directly and repeatedly calls SharedArrayPool.Rent/Return in a hot loop, so forcing these methods out-of-line adds per-call overhead — a direct, plausible mechanism for the +56% slowdown.

  Helix jobs

  Job ID	Submitted by bisection	Last status
  463a9186-bc66-4ad9-9c3b-fde3bffa82c8	✅	submitted
  d6a976cf-f83d-4307-bd38-6c55ea658f9e	✅	submitted
  50f7e4a7-e8ab-4bdc-a5cd-bdf6cc2c8b92	✅	submitted

  Per-commit measurements

  Commit	Position	Measurements
  929271670357	base, idx0, good	54.850 ns
  f079ec9d1538	idx12, good	53.390 ns
  b41622e64799	idx25, good	52.930 ns
  2e738f3b09ee	idx31, good	53.510 ns
  299a04078dc8	idx32, last-good	53.030 ns
  ce12d869155b	idx34, regressed	87.520 ns
  05311dc7564a	idx37, regressed	89.590 ns
  2119b868df7a	idx33, culprit	88.500 ns
  3522269feadf	head, idx49, regressed	86.490 ns

  Bisection details

  …[earlier output omitted]

  TION RESULT:
  - Culprit Commit: 2119b868df7afa9c480058af4912188914e64ffe
  - Commit Message: Add NoInlining to ArrayPool Rent/Return (#129319)
  - Commits Tested: 10
  - Confidence: 5
  - Commit Relevance: RELATED
  - Relevance Explanation: The commit adds [MethodImpl(MethodImplOptions.NoInlining)] to Rent and Return in SharedArrayPool.cs (and ConfigurableArrayPool.cs). The regression test RentReturnArrayPoolTests<Object>.SingleParallel with UseSharedPool: True directly and repeatedly calls SharedArrayPool.Rent/Return in a hot loop, so forcing these methods out-of-line adds per-call overhead — a direct, plausible mechanism for the +56% slowdown.
  - Baseline Value: 53.03 ns
  - Culprit Value: 88.50 ns
  - Summary: Bisected the 49-commit range with Helix jobs using regressed-count over increasing commit subsequences (order-robust). Narrowed (25,37] → (31,34] → boundary between index 32 (299a0407, good ~53 ns) and index 33 (2119b868, regressed ~88 ns), pinpointing the NoInlining commit.
  
  HELIX JOBS:
  - 463a9186-bc66-4ad9-9c3b-fde3bffa82c8
  - d6a976cf-f83d-4307-bd38-6c55ea658f9e
  - 50f7e4a7-e8ab-4bdc-a5cd-bdf6cc2c8b92
  
  MEASUREMENTS:
  - 929271670357 (base, idx0, good): 54.85 ns
  - f079ec9d1538 (idx12, good): 53.39 ns
  - b41622e64799 (idx25, good): 52.93 ns
  - 034f7f5c8439 (idx28, good): ~53 ns
  - 2e738f3b09ee (idx31, good): 53.51 ns
  - 299a04078dc8 (idx32, last-good): 53.03 ns
  - 2119b868df7a (idx33, culprit): 88.50 ns
  - ce12d869155b (idx34, regressed): 87.52 ns
  - 05311dc7564a (idx37, regressed): 89.59 ns
  - 3522269feadf (head, idx49, regressed): 86.49 ns
  

  Note: Because Helix report ordering was non-deterministic across jobs, I relied on the count of regressed values within monotonically-increasing commit subsequences (each job cleanly split into ~53 ns "good" and ~88 ns "regressed" clusters), which is order-independent and unambiguous at the boundary. Absolute values matched the reported baseline (55.13) and compare (86.21) closely, confirming the default-JIT config reproduced the regression.

  System.Buffers.Tests.RentReturnArrayPoolTests.MultipleSerial(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

  • Status: timeout

  System.Collections.Perf_Frozen.ToFrozenDictionary(Count: 512)

  • Culprit: 2119b868df7a
  • Commit Message: Add NoInlining to ArrayPool Rent/Return (Add NoInlining to ArrayPool Rent/Return #129319)
  • Relevance: RELATED
  • Explanation: The commit marks ArrayPool.Rent/Return with MethodImplOptions.NoInlining in System.Private.CoreLib. FrozenDictionary construction (ToFrozenDictionary) rents and returns pooled arrays while building its internal tables, so forcing these hot-path methods out-of-line adds real call overhead on the construction path the benchmark measures. The adjacent good→bad transition (idx32 clean, idx33 regressed) plus a plausible mechanism make this a genuine functional regression, not a layout artifact.

  Helix jobs

  Job ID	Submitted by bisection	Last status
  3a1947a3-c8d4-410a-8a94-02fe415c09d4	✅	submitted
  a873932c-7971-4d29-90fe-59975b5dd7e1	✅	submitted
  16d85aed-88c6-4e2a-a4fe-d3b7ae90c380	✅	submitted

  Per-commit measurements

  Commit	Position	Measurements
  (baseline)	baseline	23.703 μs
  929271670357	base, good	23.939 μs
  f079ec9d1538	idx12	23.985 μs
  b41622e64799	idx25	24.089 μs
  034f7f5c8439	idx28	24.140 μs
  2e738f3b09ee	idx31	23.814 μs
  299a04078dc8	idx32, last good	23.703 μs
  ce12d869155b	idx34	28.147 μs
  05311dc7564a	idx37	28.353 μs
  (culprit)	culprit	28.454 μs
  2119b868df7a	idx33, culprit	28.454 μs
  3522269feadf	head, regressed	28.627 μs

  Bisection details

  …[earlier output omitted]

  BISECTION RESULT:
  - Culprit Commit: 2119b868df7afa9c480058af4912188914e64ffe
  - Commit Message: Add NoInlining to ArrayPool Rent/Return (#129319)
  - Commits Tested: 10
  - Confidence: 5
  - Commit Relevance: RELATED
  - Relevance Explanation: The commit marks ArrayPool.Rent/Return with MethodImplOptions.NoInlining in System.Private.CoreLib. FrozenDictionary construction (ToFrozenDictionary) rents and returns pooled arrays while building its internal tables, so forcing these hot-path methods out-of-line adds real call overhead on the construction path the benchmark measures. The adjacent good→bad transition (idx32 clean, idx33 regressed) plus a plausible mechanism make this a genuine functional regression, not a layout artifact.
  - Baseline Value: 23703.2 ns
  - Culprit Value: 28453.6 ns
  - Summary: Bisected 49 commits between 929271670357 and 3522269feadf via successive Helix jobs (double-bisect: base/quarter/mid/three-quarter/head → idx28/31/34 → idx32/33), using the monotonic step property to isolate the culprit. The regression (~+18%, ~4700 ns) cleanly appears at the commit that disables inlining of ArrayPool Rent/Return, far above the noise floor.
  
  HELIX JOBS:
  - 3a1947a3-c8d4-410a-8a94-02fe415c09d4
  - a873932c-7971-4d29-90fe-59975b5dd7e1
  - 16d85aed-88c6-4e2a-a4fe-d3b7ae90c380
  
  MEASUREMENTS:
  - 929271670357 (base, good): 23939 ns
  - f079ec9d1538 (idx12): 23985 ns
  - b41622e64799 (idx25): 24089 ns
  - 034f7f5c8439 (idx28): 24140 ns
  - 2e738f3b09ee (idx31): 23814 ns
  - 299a04078dc8 (idx32, last good): 23703 ns
  - 2119b868df7a (idx33, culprit): 28454 ns
  - ce12d869155b (idx34): 28147 ns
  - 05311dc7564a (idx37): 28353 ns
  - 3522269feadf (head, regressed): 28627 ns
  

  Bisection succeeded: the +17.9% regression in Perf_Frozen<Int16>.ToFrozenDictionary(Count: 512) was introduced by commit 2119b868 ("Add NoInlining to ArrayPool Rent/Return"), which is functionally RELATED to the test.

  System.Collections.Perf_Frozen.ToFrozenDictionary(Count: 64)

  • Culprit: 2119b868df7a
  • Commit Message: Add NoInlining to ArrayPool Rent/Return (Add NoInlining to ArrayPool Rent/Return #129319)
  • Relevance: RELATED
  • Explanation: The commit marks ArrayPool.Rent/Return as NoInlining in ConfigurableArrayPool and SharedArrayPool. FrozenDictionary construction (ToFrozenDictionary) rents temporary arrays from ArrayPool.Shared while building the hashcode/bucket tables, so forcing these methods out-of-line adds real call overhead to the construction hot path. This is a plausible, direct mechanism for the +16% build-time regression.

  Helix jobs

  Job ID	Submitted by bisection	Last status
  22099954-04ef-495a-9d63-047ae54f7460	✅	submitted
  c2bed97c-58ce-4e02-915d-571f198de68c	✅	submitted
  35a3dcc2-160c-49cf-88dd-556af0602787	✅	submitted

  Per-commit measurements

  Commit	Position	Measurements
  929271670357	base, good	2.617 μs
  f079ec9d1538	q1 boundary, good	2.619 μs
  05311dc7564a	3/4 boundary, bad	3.064 μs
  2119b868df7a	culprit, bad	3.046 μs
  299a04078dc8	last good before culprit	2.618 μs
  3522269feadf	head, bad	3.002 μs
  7a1bd45a7260	good	2.599 μs
  b41622e64799	mid, good	2.614 μs
  ce12d869155b	bad	3.042 μs
  e220bf0be5ec	bad	3.034 μs

  Bisection details

  …[earlier output omitted]

  Bisection complete. The culprit adds NoInlining to ArrayPool.Rent/Return, and ToFrozenDictionary rents temporary arrays from ArrayPool during construction — a clear mechanism for the +16% regression.

  BISECTION RESULT:
  - Culprit Commit: 2119b868df7afa9c480058af4912188914e64ffe
  - Commit Message: Add NoInlining to ArrayPool Rent/Return (#129319)
  - Commits Tested: 10
  - Confidence: 5
  - Commit Relevance: RELATED
  - Relevance Explanation: The commit marks ArrayPool<T>.Rent/Return as NoInlining in ConfigurableArrayPool and SharedArrayPool. FrozenDictionary construction (ToFrozenDictionary) rents temporary arrays from ArrayPool.Shared while building the hashcode/bucket tables, so forcing these methods out-of-line adds real call overhead to the construction hot path. This is a plausible, direct mechanism for the +16% build-time regression.
  - Baseline Value: 2617.5 ns
  - Culprit Value: 3045.8 ns
  - Summary: Double-bisected the 49-commit range on Ubuntu.2204.Arm64.Perf: base/f079ec9/b41622 measured ~2615ns (good) while 05311dc/head measured ~3000-3064ns (bad), isolating the step to Q3. Further narrowing showed 299a0407 good (2617.5ns) and 2119b868 already regressed (3045.8ns), a clean ~428ns (+16%) step far above the ~46ns 2σ noise floor.
  
  HELIX JOBS:
  - 22099954-04ef-495a-9d63-047ae54f7460
  - c2bed97c-58ce-4e02-915d-571f198de68c
  - 35a3dcc2-160c-49cf-88dd-556af0602787
  
  MEASUREMENTS:
  - 929271670357 (base, good): 2616.7 ns
  - f079ec9d1538 (Q1 boundary, good): 2619.3 ns
  - b41622e64799 (mid, good): 2614.3 ns
  - 05311dc7564a (3/4 boundary, bad): 3063.6 ns
  - 3522269feadf (head, bad): 3002.4 ns
  - 7a1bd45a7260 (good): 2598.7 ns
  - 299a04078dc8 (last good before culprit): 2617.5 ns
  - e220bf0be5ec (bad): 3034.4 ns
  - 2119b868df7a (culprit, bad): 3045.8 ns
  - ce12d869155b (bad): 3041.5 ns
  

  The regression signal (~428ns, ~9σ) is well above the noise floor, so the single-sample bisection is unambiguous and no confidence cap was warranted.

  System.Collections.Perf_Frozen.ToFrozenDictionary(Count: 512)

  • Culprit: 2119b868df7a
  • Commit Message: Add NoInlining to ArrayPool Rent/Return (Add NoInlining to ArrayPool Rent/Return #129319)
  • Relevance: RELATED
  • Explanation: The commit marks ArrayPool.Rent and Return as NoInlining in ConfigurableArrayPool and SharedArrayPool. ToFrozenDictionary rents/returns temporary arrays from the shared ArrayPool while building the frozen collection, so forcing these hot-path methods out-of-line adds call overhead directly on the construction path the benchmark measures. The step change is clean and reproducible, matching the reported +19.4% magnitude.

  Helix jobs

  Job ID	Submitted by bisection	Last status
  950e5bb3-a637-422e-9423-ab27fd32675c	✅	submitted
  ee75c237-6e9f-410f-80e4-b75e044ab556	✅	submitted
  52aa8216-4d83-4d3d-a0fc-a2272642eb5d	✅	submitted

  Per-commit measurements

  Commit	Position	Measurements
  929271670357	base, good	26.738 μs
  f079ec9d1538	idx12, good	26.769 μs
  b41622e64799	idx25, good	26.642 μs
  034f7f5c8439	idx28, good	26.881 μs
  2e738f3b09ee	idx31, good	26.725 μs
  299a04078dc8	idx32, last good	26.947 μs
  ce12d869155b	idx34, bad	32.006 μs
  05311dc7564a	idx37, bad	31.987 μs
  2119b868df7a	idx33, culprit	31.400 μs
  3522269feadf	head, bad	31.952 μs

  Bisection details

  …[earlier output omitted]

  ","abbrevSha":"2...

  Bisection complete. The culprit adds NoInlining to ArrayPool.Rent/Return — and ToFrozenDictionary uses ArrayPool for temporary buffers during construction, so this directly affects the test's hot path. Clean single-commit result with a ~4500 ns step (≫2σ=508).

  BISECTION RESULT:
  - Culprit Commit: 2119b868df7afa9c480058af4912188914e64ffe
  - Commit Message: Add NoInlining to ArrayPool Rent/Return (#129319)
  - Commits Tested: 10
  - Confidence: 5
  - Commit Relevance: RELATED
  - Relevance Explanation: The commit marks ArrayPool<T>.Rent and Return as NoInlining in ConfigurableArrayPool and SharedArrayPool. ToFrozenDictionary rents/returns temporary arrays from the shared ArrayPool while building the frozen collection, so forcing these hot-path methods out-of-line adds call overhead directly on the construction path the benchmark measures. The step change is clean and reproducible, matching the reported +19.4% magnitude.
  - Baseline Value: 26946.8 ns   (last good, 299a04078dc8)
  - Culprit Value: 31399.8 ns    (culprit, 2119b868df7a)
  - Summary: Bisected base..head (49 commits) via quadrant sampling: good through idx25 (b41622e6=26641.6) and idx31 (2e738f3b=26725.0), bad at idx34 (ce12d869=32005.7); then tested idx32 (299a0407=26946.8, good) and idx33 (2119b868=31399.8, bad), isolating the first bad commit. Default tiered-JIT config; absolute values align with reported baseline 26.93 / compare 32.16.
  
  HELIX JOBS:
  - 950e5bb3-a637-422e-9423-ab27fd32675c
  - ee75c237-6e9f-410f-80e4-b75e044ab556
  - 52aa8216-4d83-4d3d-a0fc-a2272642eb5d
  
  MEASUREMENTS:
  - 929271670357 (base, good): 26737.6 ns
  - f079ec9d1538 (idx12, good): 26768.9 ns
  - b41622e64799 (idx25, good): 26641.6 ns
  - 034f7f5c8439 (idx28, good): 26880.7 ns
  - 2e738f3b09ee (idx31, good): 26725.0 ns
  - 299a04078dc8 (idx32, last good): 26946.8 ns
  - 2119b868df7a (idx33, culprit): 31399.8 ns
  - ce12d869155b (idx34, bad): 32005.7 ns
  - 05311dc7564a (idx37, bad): 31986.6 ns
  - 3522269feadf (head, bad): 31952.4 ns
  

  System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 32)

  • Culprit: 2119b868df7a
  • Commit Message: Add NoInlining to ArrayPool Rent/Return (Add NoInlining to ArrayPool Rent/Return #129319)
  • Relevance: RELATED
  • Explanation: The commit adds [MethodImpl(NoInlining)] to ArrayPool.Rent/Return in ConfigurableArrayPool and SharedArrayPool. BinaryWriter.Write(char[]) rents a byte buffer from the shared ArrayPool on its encoding hot path, so forcing these tiny methods to remain out-of-line adds call overhead directly to the tested code path. The measured step change (26.42 ns → 33.20 ns, +25.7%) occurs exactly at this commit and matches the reported regression magnitude.

  Helix jobs

  Job ID	Submitted by bisection	Last status
  45c44565-28be-43ec-a15c-0fc3042c158b	✅	submitted
  67eba11c-e6ce-433d-b10d-620f23ba0529	✅	submitted
  5a341ca1-2a95-43f3-a4f6-e405707d7868	✅	submitted

  Per-commit measurements

  Commit	Position	Measurements
  929271670357	base, good	26.373 ns
  f079ec9d1538	q1, good	26.376 ns
  05311dc7564a	q3, regressed	33.061 ns
  034f7f5c8439	good	26.448 ns
  2119b868df7a	culprit	33.197 ns
  299a04078dc8	last good	26.424 ns
  2e738f3b09ee	good	26.069 ns
  3522269feadf	head, regressed	33.085 ns
  b41622e64799	mid, good	26.334 ns
  ce12d869155b	regressed	33.222 ns

  Bisection details

  …[earlier output omitted]

  └ {"index":null,"sha":"2119b868df7afa9c480058af4912188914e64ffe","abbrevSha":"2...

  Confirmed. The culprit adds NoInlining to ArrayPool.Rent/Return. BinaryWriter.Write(char[]) rents a byte buffer from ArrayPool on its encoding hot path, so preventing inlining of Rent/Return directly adds call overhead — a clear mechanism. Measured values (26.42 → 33.20, +25.7%) match the reported regression exactly.

  BISECTION RESULT:
  - Culprit Commit: 2119b868df7afa9c480058af4912188914e64ffe
  - Commit Message: Add NoInlining to ArrayPool Rent/Return (#129319)
  - Commits Tested: 12
  - Confidence: 5
  - Commit Relevance: RELATED
  - Relevance Explanation: The commit adds [MethodImpl(NoInlining)] to ArrayPool<T>.Rent/Return in ConfigurableArrayPool and SharedArrayPool. BinaryWriter.Write(char[]) rents a byte buffer from the shared ArrayPool on its encoding hot path, so forcing these tiny methods to remain out-of-line adds call overhead directly to the tested code path. The measured step change (26.42 ns → 33.20 ns, +25.7%) occurs exactly at this commit and matches the reported regression magnitude.
  - Baseline Value: 26.424 ns
  - Culprit Value: 33.197 ns
  - Summary: Started with a 5-commit spread across the 49-commit range, localized the step to Q3, then bisected: index 31 (2e738f3) good, index 34 (ce12d869) bad; index 32 (299a0407) good at 26.42 ns, index 33 (2119b868) regressed to 33.20 ns, isolating the culprit.
  
  HELIX JOBS:
  - 45c44565-28be-43ec-a15c-0fc3042c158b
  - 67eba11c-e6ce-433d-b10d-620f23ba0529
  - 5a341ca1-2a95-43f3-a4f6-e405707d7868
  
  MEASUREMENTS:
  - 929271670357 (base, good): 26.373 ns
  - f079ec9d1538 (Q1, good): 26.376 ns
  - b41622e64799 (mid, good): 26.334 ns
  - 05311dc7564a (Q3, regressed): 33.061 ns
  - 3522269feadf (head, regressed): 33.085 ns
  - 034f7f5c8439 (good): 26.448 ns
  - 2e738f3b09ee (good): 26.069 ns
  - ce12d869155b (regressed): 33.222 ns
  - 299a04078dc8 (last good): 26.424 ns
  - 2119b868df7a (culprit): 33.197 ns
  

  🤖 Proposed Next Actions

  Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

  Verified regression attributed to commit 2119b86 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

  #	Action	Status	Details
  1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for 2119b868df7a
  2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
  3	Ping owners	⏭️ Skipped (dry-run)	ping jakobbotsch

  🦆 Pre-bisection Re-analysis (advisory)

  ⚠️ Disputes the pre-bisection plan — advisory only, not applied

  Flags: wasted-bisection-slot

  Rationale: The four actionable findings (MultipleSerial changeConf 5/5, three ToFrozenDictionary variants changeConf 3/5, all non-noise and stack-relevant) reasonably justify bisecting their four corresponding candidates, but two queued tests (SingleParallel UseSharedPool:True and WriteAsciiCharArray) have no supporting per-finding signal at all and would burn ~60+ Helix minutes each without evidence of a regression. Drop those two unless signal is added.

  Suggested drop from bisection: System.Buffers.Tests.RentReturnArrayPoolTests<Object>.SingleParallel(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: True), System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 32) (advisory — not applied)

  This critique is advisory in the current configuration and did not change which tests were bisected.

  🦆 Re-analysis (advisory)

  ⚠️ Disputes the analysis — advisory only, not applied

  Flags: overconfident-change, bimodal-history

  Rationale: The culprit 2119b86 is well-supported (four benchmarks' bisections converged on it with RELATED relevance, no alignment warnings), so I accept both the culprit and the strong finding confidence, but a maxed-out 5/5 regression confidence is not justified when bimodal history is flagged and the headline MultipleSerial benchmark's own bisection timed out, so I lower change confidence to 4.

  Suggested confidence: Change Confidence → 4/5 (advisory — not applied)

  This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

  ---

  Generated by PerfTriageAgent on 2026-07-23 14:43 UTC