[Perf] Linux/x64: 2 Regressions on 4/18/2026 4:02:25 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	38af44d518908fa536ef289d62c73706769d18ab
Compare	b70c35ed8a2e7ae0d91de76f4f5d26c2e7d2c6cd
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSetReverse<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Array - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	195.26 ns	258.86 ns	1.33	0.20	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSetReverse<String>*'

Details

System.Collections.IndexerSetReverse<String>.Array(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	38af44d518908fa536ef289d62c73706769d18ab
Compare	b70c35ed8a2e7ae0d91de76f4f5d26c2e7d2c6cd
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Array - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	216.77 ns	283.52 ns	1.31	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet<String>*'

Details

System.Collections.IndexerSet<String>.Array(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: *Note on Noise Classification: Both tests are statistically classified as noise by the adaptive threshold method. However, this is due to single extreme outliers (~15270–15397 ns) in the historical data that inflate σ to ~901–903 and CV to ~332–346%, making the adaptive threshold (663–691%) absurdly high. The recent-before series was stable at ~195/217 ns respectively, and the compare values represent a clear ~30% step change. The statistical classification should not be trusted without fir

Finding Confidence: 3/5 (analysis accuracy)
Regression Confidence: 4/5 (likelihood of true regression)
⚠️ Assessment: Likely noise - recommend closing or monitoring for recurrence.

📊 Full Analysis (click to expand)

Issue Overview

• Issue: #71660 - [Perf] Linux/x64: 2 Regressions on 4/18/2026 4:02:25 PM +00:00
• Date: 2026-04-18 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: linux/x64
• Commits: 38af44d51890 → b70c35ed8a2e (9 commits)
• Labels: untriaged, perf-regression, arch-x64, os-linux, branch-refs/heads/main, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Historical Data: Available (279–283 runs) — WARNING: Historical statistics severely contaminated by extreme outliers (~15270–15397 ns), rendering adaptive thresholds unusable. Recent-before series was stable.

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
IndexerSetReverse<String>.Array(Size: 512)	195.26 ns	258.86 ns	+32.6%	261.25	903.03	283	-0.003	691.3%	Y*	Step change from stable ~195 baseline; stats distorted by outlier
IndexerSet<String>.Array(Size: 512)	216.77 ns	283.52 ns	+30.8%	271.62	901.23	279	0.013	663.6%	Y*	Step change from stable ~217 baseline; stats distorted by outlier

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

*Note on Noise Classification: Both tests are statistically classified as noise by the adaptive threshold method. However, this is due to single extreme outliers (~15270–15397 ns) in the historical data that inflate σ to ~901–903 and CV to ~332–346%, making the adaptive threshold (663–691%) absurdly high. The recent-before series was stable at ~195/217 ns respectively, and the compare values represent a clear ~30% step change. The statistical classification should not be trusted without first cleaning the outliers from the historical baseline.

Finding Confidence: 3/5 — Adequate run count (279–283 runs) but historical statistics are unreliable due to extreme outlier contamination. The visual step change is apparent but cannot be statistically confirmed with current data quality.

Confirmed Regressions

System.Collections.IndexerSetReverse<String>.Array(Size: 512) (+32.6%)

• Delta: 195.26 ns → 258.86 ns (+32.6%)
• Historical Context: μ=261.25 (σ=903.03, n=283); z-score=-0.003; adaptive threshold=691.3% — statistics invalidated by outlier. Recent-before mean ~195.41 ns was stable.
• Assessment: Apparent step change from stable ~195 ns to ~259 ns. Statistically classified as noise only due to extreme outlier contamination.
• Source: IndexerSetReverse.cs#L27-L34 — iterates array in reverse setting array[i] = default (null for String/reference type, requiring write barrier)
• Finding Confidence: 3/5 — Historical data volume is good but statistics are contaminated; step change is visually clear
• Regression Confidence: 4/5 — Strong signal (+32.6%), direct code path match (WriteBarrier for reference-type array stores), small commit range (9 commits), surgical likely cause
• Likely Cause: 6f82234fa8f5 - "Reapply 'Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall'"
  • Affected Areas: JIT / CoreCLR / Write Barriers / Array Stores
  • Rationale: This commit inlines the CORINFO_HELP_ARRADDR_ST helper call and removes the WriteBarrier FCall for array element stores. The benchmark sets array[i] = default for String (a reference type), which requires a write barrier on every store. Changing from an FCall-based write barrier to an inlined implementation directly impacts this hot path. The commit was authored by EgorBo and is a reapply of a previously reverted change (Revert "Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall" #126530, original Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall #117583). The timing aligns perfectly — committed at 2026-04-18T13:21:06, before the regression date of 2026-04-18T16:02:25.
  • Candidates Considered:
    • 6f82234fa8f5 - Primary: Directly modifies write barrier / array store helpers in JIT (29 files, compiler.h, gentree.cpp, importercalls.cpp, morph.cpp, CastHelpers.cs, Array.CoreCLR.cs, asm helpers)
    • b70c35ed8a2e - Improves SIMD/FP costing in JIT; unlikely since benchmark is integer/reference-type array stores, not SIMD/FP
    • 29bafa7bc4e8 - Adds F16C instruction support; unrelated to array stores
• Verification Status: Not verified — bisection recommended to confirm

System.Collections.IndexerSet<String>.Array(Size: 512) (+30.8%)

• Delta: 216.77 ns → 283.52 ns (+30.8%)
• Historical Context: μ=271.62 (σ=901.23, n=279); z-score=0.013; adaptive threshold=663.6% — statistics invalidated by outlier. Recent-before mean ~217.43 ns was stable. Note: last after-value (215.57) reverted to baseline levels, suggesting the regression may be transient.
• Assessment: Apparent step change from stable ~217 ns to ~283 ns. One after-run reverted, which may indicate instability.
• Source: IndexerSet.cs#L35-L42 — iterates array forward setting array[i] = default (null for String/reference type, requiring write barrier)
• Finding Confidence: 3/5 — Same outlier contamination issue; additionally, one after-run reverted to baseline, introducing uncertainty
• Regression Confidence: 4/5 — Strong signal (+30.8%), same code path as companion test, corroborating evidence from IndexerSetReverse. Slight concern about the reverted after-value.
• Likely Cause: 6f82234fa8f5 - "Reapply 'Inline CORINFO_HELP_ARRADDR_ST helper call, remove WriteBarrier FCall'"
  • Affected Areas: JIT / CoreCLR / Write Barriers / Array Stores
  • Rationale: Same as IndexerSetReverse above — both benchmarks perform array element stores for reference types (String), which require write barriers. The forward-iteration variant exercises the same JIT-inlined write barrier path. The corroboration between both tests strongly supports this attribution.
  • Candidates Considered: Same as IndexerSetReverse above.
• Verification Status: Not verified — bisection recommended to confirm

Open Questions

• Historical Data: Severely contaminated by extreme outliers (~15270–15397 ns, ~80x normal values) in both test series. This renders σ, z-score, and adaptive thresholds unreliable. Outlier scrubbing or winsorization is recommended for future analysis.
• Confidence Limitations: Statistical noise classification conflicts with visually apparent step change. The regression cannot be statistically confirmed using current methodology without outlier removal.
• Transient Behavior: For IndexerSet (non-Reverse), the last after-value (215.57 ns) reverted to baseline levels. This may indicate the regression is intermittent or environment-dependent.
• Verification Needed: Bisection on commits 38af44d51890..b70c35ed8a2e (9 commits) to isolate whether 6f82234fa8f5 is the culprit.
• Bisection Status: Not performed

Recommended Actions

1. Bisection: Run bisection on commits 38af44d51890..b70c35ed8a2e using RunWithBuildAtHash to confirm 6f82234fa8f5 as the culprit. Given only 9 commits, testing the baseline, the suspected commit (6f82234), and the compare should suffice.
2. Source Review: Review the inlined write barrier codegen produced by 6f82234fa8f5 for reference-type array stores. The ~30% regression on a simple array[i] = default loop suggests the inlined path may be generating suboptimal code compared to the previous FCall helper.
3. Historical Data Cleanup: Investigate and scrub the extreme outlier runs (~15270–15397 ns) from both test series to restore meaningful statistical analysis for future regressions.
4. Monitor: Track subsequent runs to determine if the IndexerSet (non-Reverse) reversion to baseline was an anomaly or indicates intermittent behavior.

Structured Summary (JSON)

{
  "schemaVersion": "perf-triage-v1",
  "issue": 71660,
  "baselineCommit": "38af44d518908fa536ef289d62c73706769d18ab",
  "compareCommit": "b70c35ed8a2e7ae0d91de76f4f5d26c2e7d2c6cd",
  "commitRangeSize": 9,
  "historicalContext": {
    "runsExamined": 579,
    "missing": false,
    "notes": "Historical statistics severely contaminated by extreme outliers (~15270-15397 ns) inflating CV to 332-346%. Adaptive thresholds (663-691%) are unusable. Recent-before series was stable at ~195/217 ns. Visual step change is apparent but not statistically confirmable with current data."
  },
  "regressions": [
    {
      "test": "System.Collections.IndexerSetReverse<String>.Array(Size: 512)",
      "baseline": 195.258,
      "compare": 258.859,
      "percentDelta": 32.57,
      "histMean": 261.246,
      "histStdDev": 903.027,
      "zScore": -0.003,
      "isNoise": true,
      "likelyCommit": "6f82234fa8f5",
      "findingConfidence": 3,
      "findingConfidenceRationale": "Good data volume (283 runs) but statistics invalidated by extreme outlier; visual step change clear but unconfirmable statistically",
      "regressionConfidence": 4,
      "regressionConfidenceRationale": "Strong +32.6% step change from stable baseline, direct write barrier code path match, small commit range (9), corroborated by companion test",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    },
    {
      "test": "System.Collections.IndexerSet<String>.Array(Size: 512)",
      "baseline": 216.773,
      "compare": 283.522,
      "percentDelta": 30.79,
      "histMean": 271.619,
      "histStdDev": 901.225,
      "zScore": 0.013,
      "isNoise": true,
      "likelyCommit": "6f82234fa8f5",
      "findingConfidence": 3,
      "findingConfidenceRationale": "Good data volume (279 runs) but statistics invalidated by extreme outlier; one after-run reverted to baseline adding uncertainty",
      "regressionConfidence": 4,
      "regressionConfidenceRationale": "Strong +30.8% step change from stable baseline, direct write barrier code path match, corroborated by companion test; slight concern about reverted after-value",
      "verified": false,
      "bisected": false,
      "bisectionRange": null
    }
  ],
  "nextActions": [
    "Bisect 9 commits to confirm 6f82234fa8f5 (WriteBarrier inlining) as culprit",
    "Review inlined write barrier codegen for reference-type array stores",
    "Scrub extreme outliers from historical data for both tests",
    "Monitor subsequent runs for IndexerSet stability"
  ],
  "generationNotes": "Statistical noise classification driven by extreme outlier contamination (single ~15k ns values in series normally at ~195-283 ns). Both AnalyzeHistoricalData and adaptive thresholds classify as noise, but the visual evidence of a ~30% step change from stable baselines, combined with direct code path relevance of commit 6f82234fa8f5, warrants treating this as a likely genuine regression pending verification."
}

---

Generated by PerfTriageAgent on 2026-04-21 09:36 UTC

[DrewScoggins]

Seems related to #126547

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[DrewScoggins]

Regressions:

• [Perf] Windows/x64: 7 Regressions on 4/18/2026 4:02:25 PM +00:00 perf-autofiling-issues#71669
• [Perf] Linux/arm64: 2 Regressions on 4/18/2026 5:58:25 PM +00:00 #127330