[Perf] Linux/arm64: 4 Regressions on 5/12/2026 11:35:36 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	6820c801e8b84f3fc9f96ecb6e0838b829345951
Compare	52090b9f07a14ec5db3a19b379548f469fefcc27
Diff	Diff
Configs	CompilationMode:tiered, R2RType:r2r_interpreter, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Double>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IndexOfMax - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	5.12 μs	7.34 μs	1.44	0.05	False
IndexOfMax - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	217.05 ns	302.22 ns	1.39	0.14	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Double>*'

Details

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Double>.IndexOfMax(BufferLength: 3079)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Double>.IndexOfMax(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	6820c801e8b84f3fc9f96ecb6e0838b829345951
Compare	52090b9f07a14ec5db3a19b379548f469fefcc27
Diff	Diff
Configs	CompilationMode:tiered, R2RType:r2r_interpreter, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
IndexOfMax - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	2.56 μs	3.67 μs	1.43	0.12	False
IndexOfMax - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	110.52 ns	147.74 ns	1.34	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>*'

Details

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>.IndexOfMax(BufferLength: 3079)

ETL Files

Histogram

JIT Disasms

System.Numerics.Tensors.Tests.Perf_NumberTensorPrimitives<Single>.IndexOfMax(BufferLength: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: All four regressions share the same root cause — they are all TensorPrimitives.IndexOfMax<T> benchmarks differing only in element type (Double/Single) and buffer size (128/3079). The consistent magnitude (~33-44%) across all variants points to a fundamental change in the SIMD/vectorization code path rather than a buffer-size-specific issue. Bisection skipped — autofile run configuration is not supported by the bisection harness.

Finding Confidence: 4/5 (analysis accuracy)
Regression Confidence: 4/5 (likelihood of true regression)

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #72942 - [Perf] Linux/arm64: 4 Regressions on 5/12/2026 11:35:36 PM +00:00
• Date: 2026-05-12 | Queue: AzureLinux.3.Cobalt.Arm64.Perf | OS/Arch: Linux/arm64
• Commits: 6820c801e8b8 → 52090b9f07a1 (19 commits)
• Labels: untriaged, perf-regression, os-linux, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, r2rtype-r2r_interpreter, runkind-micro
• Historical Data: Available (96-97 runs per test, 414 total runs examined)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
Perf_NumberTensorPrimitives<Double>.IndexOfMax(BufferLength: 3079)	5117.15 ns	7343.94 ns	+43.52%	5097.07	90.04	96	24.95	5%	N	Step change, extremely stable
Perf_NumberTensorPrimitives<Double>.IndexOfMax(BufferLength: 128)	217.05 ns	302.22 ns	+39.24%	213.31	15.25	97	5.83	14.3%	N	Step change, stable core
Perf_NumberTensorPrimitives<Single>.IndexOfMax(BufferLength: 3079)	2563.44 ns	3671.02 ns	+43.21%	2510.86	190.51	97	6.09	15.18%	N	Step change, stable core
Perf_NumberTensorPrimitives<Single>.IndexOfMax(BufferLength: 128)	110.52 ns	147.74 ns	+33.67%	109.25	6.02	97	6.40	11.01%	N	Step change, stable core

Legend: μ=mean (ns), σ=stddev, n=prior runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Excellent historical data (96-97 runs per test), clear statistical signal with z-scores ranging from 5.8 to 25.0, and sufficient post-regression runs (6-7) showing tight clustering.

Confirmed Regressions

All 4 IndexOfMax Tests (+33% to +44%)

All four regressions share the same root cause — they are all TensorPrimitives.IndexOfMax<T> benchmarks differing only in element type (Double/Single) and buffer size (128/3079). The consistent magnitude (~33-44%) across all variants points to a fundamental change in the SIMD/vectorization code path rather than a buffer-size-specific issue.

• Delta: +33.67% to +43.52% across all four test variants

• Historical Context: All tests show extremely strong z-scores (5.8 to 25.0); historical distributions are stable with tight clustering before and after the change

• Assessment: Clear step change. Near-zero variance in post-regression values rules out environmental flakiness — this is a deterministic regression.

• Finding Confidence: 4/5 - Excellent historical data quality, sufficient runs, clear signal

• Regression Confidence: 4/5 - Extreme z-scores, consistent across all 4 variants, two highly relevant commits in range

• Likely Cause: 06c12228774a - "Arm64: Fix loads and stores for stack allocated vectors and masks (Arm64: Fix loads and stores for stack allocated vectors and masks #128037)"

  • Affected Areas: JIT / ARM64 code generation / SIMD vector operations
  • Rationale: This commit directly modifies src/coreclr/jit/emitarm64.cpp, the ARM64 JIT emitter, specifically fixing loads and stores for stack-allocated vectors and masks. The IndexOfMax benchmark relies heavily on SIMD vector operations through TensorPrimitives.IndexOfMax<T>, and this regression is ARM64-specific. A correctness fix for vector/mask stack operations could change code generation patterns (e.g., spilling vectors to stack more frequently or using different instruction sequences), leading to a consistent performance regression. The ~33-44% magnitude across both element types and buffer sizes is consistent with a fundamental change in how vectors are loaded/stored.
  • Candidates Considered:
    • 06c12228774a - Primary suspect: ARM64 JIT emitter change for vector/mask loads and stores — directly affects SIMD codegen on the regressing platform
    • 7b09f2aa8aac - Secondary suspect: "Ensure gtInitializeStoreNode consistently handles SIMD/Mask stores for all platforms" by tannergooding — modifies src/coreclr/jit/gentree.cpp for SIMD/Mask store node initialization, cross-platform change that could also affect ARM64 vector codegen
    • 7d32d804531f - Unlikely: wasm/coreclr test enablement, no codegen changes
    • 5e21845e3c25 - Unlikely: async thunk compilation in composite mode (NativeAOT), not related to micro benchmark JIT path

• Verification Status: Not verified. Bisection or RunWithBuildAtHash recommended to distinguish between the two SIMD-related commits (06c1222 and 7b09f2a).

Benchmark Source

The IndexOfMax benchmark calls TensorPrimitives.IndexOfMax<T>(_source1) (source), which exercises SIMD-accelerated search over numeric tensors.

Open Questions

• Multiple Candidates: Two commits (06c1222 and 7b09f2a) both modify SIMD/vector-related JIT code. While 06c1222 is more likely due to being ARM64-specific, 7b09f2a (cross-platform SIMD store node handling) could also contribute.
• Verification Needed: Bisection between the two SIMD commits would definitively identify the culprit. The commit range is small (19 commits) making this straightforward.
• Bisection Status: Not performed.

Recommended Actions

1. Bisect between SIMD commits: Run RunWithBuildAtHash comparing 6820c801e8b8 vs 7b09f2aa8aac vs 06c12228774a to isolate whether the regression comes from the gentree.cpp SIMD change or the emitarm64.cpp change.
2. Review ARM64 codegen: Examine the JIT disassembly for IndexOfMax before and after commit 06c1222 to identify what instruction sequence changes occurred.
3. Engage JIT team: Notify the ARM64 JIT team (snickolls-arm, tannergooding) about the regression to determine if the correctness fix in Arm64: Fix loads and stores for stack allocated vectors and masks #128037 has a known performance trade-off or if the regression is unintentional.

---

Generated by PerfTriageAgent on 2026-05-14 08:45 UTC

[DrewScoggins]

Seems related to #128037

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics-tensors
See info in area-owners.md if you want to be subscribed.