[Perf] Linux/arm64: 3 Regressions on 7/5/2026 2:44:21 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	986fba00d9f89d2dfb8ac440bf37727a18828f64
Compare	8f365c934088b8ae21a4dfce8073da0d3eb22a54
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
Split - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	23.91 ns	29.00 ns	1.21	0.16	False
Split - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	24.01 ns	28.85 ns	1.20	0.17	False
Split_Csv - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	73.31 μs	80.68 μs	1.10	0.25	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Details

System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None)

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: RemoveEmptyEntries)

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.Split_Csv(testName: "Long Text", lines: ["Use Case ID/Name,Mission Area,Agency,Bureau/Component/Office,Topic Area,Intended purpose and expected benefits of use case,Description of AI system's outputs,Current stage of development,Rights-impacting or Safety-impacting,Date Initiated,Development/Acquisition Date,Date Implemented,Date Retired,Open-source link to project code (if available),Supporting a High Impact Service Provider (HISP),"If yes to supporting a HISP, which one?",Public-facing service in HISP,Developed under contract(s) or in-house,Agency-owned Data Description,Demographic variables used in model features", "USDA-001: Repair Spend,"REE: Research, Education, and Economics",ARS: Agricultural Research Service,Administrative and Financial Management,Mission-Enabling (Internal Agency Support),"The intended purpose of this model is to review financial documents and then classify each expense as money spent on ""facility repairs"" or ""not facility repairs"". The expected benefits include reduction of manual hours identifying the types of transactions.","The output of the model is a recommendation of which financial transactions should be identified as ""repair"" expenses.","Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,10/1/2019,10/1/2019,6/9/2020,,,No,,,Developed with both contracting and in-house resources,"Approximately 14,000 financial transactions were used to train the model and finetune its parameters. Approximately 3,500 financial transactions were used to test the performance of the final model.",None;", "USDA-002: ARS Project Mapping ,"REE: Research, Education, and Economics",ARS: Agricultural Research Service,Office of National Programs,Science & Space,"The intended purpose of this model is to process research plans from various research program portfolios in the Agricultural Research Service (ARS) to find patterns and opportunities between projects. The expected benefits include decreasing the time that humans would spend to manually read, pull out key terms, and group the projects by topic. The model may also find patterns that a human might miss.","The model outputs groups of similar projects and project terms. The output includes metrics (silhouette scores, term rank, importance scores) that show how well the projects and terms in a group match.","Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,1/1/2020,1/1/2021,5/1/2022,,,No,,,Developed with contracting resources,"The data is a collection of project plans written by scientists, roughly 600 text documents. The texts are related to publicly available 5-year action plans. ",None;", "USDA-003: NAL Automated Indexing,"REE: Research, Education, and Economics",ARS: Agricultural Research Service,National Agricultural Library,Science & Space,This system automatically assigns word tags to agricultural research articles from a controlled list of terms provided by the National Agricultural Library Thesaurus (NALT). The tags can be used to look up and retrieve articles. Using these tags benefits users by making it easier to find the content they are looking for.,The model outputs terms to use as search tags that are specific to the article that the model analyzed.,"Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,6/1/2011,1/1/2012,6/1/2012,,,No,,,Developed with both contracting and in-house resources,The data is a collection of text scripts from publishers that have undergone quality assurance and quality control processing.,None;", "USDA-004: Predictive Modeling of Invasive Pest Species,MRP: Marketing and Regulatory Programs,APHIS: Animal and Plant Health Inspection Service,Plant Protection and Quarantine,Mission-Enabling (Internal Agency Support),"The purpose of the model is to check how likely it is for imported agricultural products from other countries to have pests. Benefits include more reliable discovery and quarantine of invasive pests, preventing pest invasion and making trade safer.",The model outputs are a prediction of whether a product carries an invasive species and what invasive species category the pest belongs to.,"Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,7/1/2015,7/1/2015,5/1/2018,,,No,,,Developed in-house,Inspection data was collected from Plant Protection and Quarantine (PPQ) and Customs and Border Protection (CBP). Quality control of the data was conducted by data analysts. There was no data augmention performed. The data is structured and has several million records with more than 50 numerical and categorical variables. This data is not publicly available. ,None;", ...])

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None) — caused by commit e9914e4 ("Optimize string.Split(char, ...) (#125379)"). 2 tests bisected to related culprit commits.

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: e9914e44b44095cfab74d577d438c29024e0cf55

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 4 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ⚠️ disputes (overconfident, weak-per-finding-evidence)

📊 Full Analysis (click to expand)

Summary: Three String.Split tests regressed +10–21% on Linux/arm64, attributed to commit e9914e4 ("Optimize string.Split(char, ...)") which directly modified String.Manipulation.cs with 231 lines of changes to the Split code path. Finding confidence 3/5 (borderline statistical classification due to inflated historical variance), regression confidence 4/5 (direct code path match, step change pattern, small 4-commit range).

Issue Overview

• Issue: [Perf] Linux/arm64: 3 Regressions on 7/5/2026 2:44:21 PM +00:00 #130418 - [Perf] Linux/arm64: 3 Regressions on 7/5/2026 2:44:21 PM +00:00
• Date: 2026-07-05 | Queue: AzureLinux.3.Cobalt.Arm64.Perf | OS/Arch: linux/arm64
• Commits: 986fba00d9f8 → 8f365c934088 (4 commits)
• Labels: untriaged, perf-regression, os-linux, branch-refs/heads/main, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime Stack: Desktop + CoreCLR (implicit)
• Historical Data: Partial (300 runs for Split tests with inflated variance due to regime shift; 0 runs for Split_Csv)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
Perf_String.Split(...options: None)	23.91 ns	29.00 ns	+21.3%	23.891	2.427	300	2.105	20.32%	N	Borderline - step change, barely exceeds threshold
Perf_String.Split(...options: RemoveEmptyEntries)	24.01 ns	28.85 ns	+20.2%	24.006	2.372	300	2.042	19.76%	N	Borderline - step change, mirrors sibling test
Perf_String.Split_Csv("Long Text", ...)	73.31 μs	80.68 μs	+10.0%	—	—	0	—	5%	N	Unknown - no historical data

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 3/5 - Historical data available (300 runs) but inflated variance from regime shift and low outliers raises adaptive threshold. Split_Csv has zero historical runs. Statistical classifications are borderline/unknown rather than confirmed.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
e9914e44b440	src/libraries/System.Private.CoreLib/src/System/String.Manipulation.cs, SpanHelpers.Packed.cs	Relevant	Direct String.Split optimization - 231 additions to Split code path
8f365c934088	src/coreclr/gc/handletable.inl	Relevant	ARM64 VolatileStore in GC handle - adds memory barrier but unlikely in hot path
58dfec78ed8d	src/libraries/System.Private.CoreLib/src/System/String.Manipulation.cs	Relevant	Minor refactor of ThrowSubstringArgumentOutOfRange to static - not in Split path
2210dd5a5133	eng/build.sh, docs/	Shared	Build script change only - no runtime impact

Confirmed Regressions

System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None) (+21.3%)

• Delta: 23.91 ns → 29.00 ns (+21.3%)
• Historical Context: μ=23.891 (σ=2.427, n=300); z-score=2.105; adaptive threshold=20.32%
• Assessment: Step change. After values consistently ~28.7–29.3 ns vs recent baseline ~23.9 ns. The high σ and CV (10.2%) are artifacts of a historical regime shift (~26 ns → ~23.9 ns) and recurring low outliers; recent pre-regression baseline is stable.
• Finding Confidence: 3/5 - Adequate historical data but borderline statistical classification due to inflated variance from regime shift
• Regression Confidence: 4/5 - Direct code path hit (String.Split optimization), consistent step change in post-regression values, small commit range (4 commits), test uses char array Split which was directly modified
• Likely Cause: e9914e44b440 - "Optimize string.Split(char, ...) (Optimize string.Split(char, ...) #125379)"
  • Affected Areas: String.Split / BCL / System.Private.CoreLib
  • Stack Relevance: Relevant
  • Rationale: This commit rewrites the string.Split(char, ...) implementation with 231 lines added to String.Manipulation.cs. The PR describes ARM64 improvements for the single-char overload with rare/no separators, but this benchmark uses arr: [' '] (char array overload) on a string with NO matching separators. The refactoring likely changed the code path for char-array-based splits, introducing overhead on ARM64 in this specific scenario. The commit timestamp (2026-07-05T14:44:21) exactly matches the regression date.
  • Candidates Considered:
    • e9914e44b440 - Direct String.Split code change, timestamp matches exactly
    • 8f365c934088 - VolatileStore in GC handles adds ARM64 memory barrier, but not in String.Split hot path
    • 58dfec78ed8d - Static refactor of ThrowSubstringArgumentOutOfRange, not in Split execution path
• Verification Status: Not verified

System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: RemoveEmptyEntries) (+20.2%)

• Delta: 24.01 ns → 28.85 ns (+20.2%)
• Historical Context: μ=24.006 (σ=2.372, n=300); z-score=2.042; adaptive threshold=19.76%
• Assessment: Step change. Mirrors the options:None sibling test exactly - same magnitude, same step change pattern. Almost certainly shares the same root cause.
• Finding Confidence: 3/5 - Same inflated variance issue as sibling test
• Regression Confidence: 4/5 - Paired with sibling test (corroboration), same direct code path match
• Likely Cause: e9914e44b440 - "Optimize string.Split(char, ...) (Optimize string.Split(char, ...) #125379)"
  • Affected Areas: String.Split / BCL / System.Private.CoreLib
  • Stack Relevance: Relevant
  • Rationale: Same commit and mechanism as the options:None test above. Both char-array Split variants are affected identically.
  • Candidates Considered: Same as above
• Verification Status: Not verified

System.Tests.Perf_String.Split_Csv(testName: "Long Text", ...) (+10.0%)

• Delta: 73.31 μs → 80.68 μs (+10.0%)
• Historical Context: No historical data available (0 runs)
• Assessment: Unknown classification due to missing historical data. The +10% delta exceeds the fixed 5% threshold. Split_Csv exercises string splitting with comma separators on multi-line CSV data - likely exercises the same modified Split code paths.
• Finding Confidence: 2/5 - No historical data available to validate signal
• Regression Confidence: 3/5 - Above 5% threshold, same commit range as confirmed Split regressions, but no historical context to assess noise level
• Likely Cause: e9914e44b440 - "Optimize string.Split(char, ...) (Optimize string.Split(char, ...) #125379)"
  • Affected Areas: String.Split / BCL / System.Private.CoreLib
  • Stack Relevance: Relevant
  • Rationale: Split_Csv uses String.Split with char separators (commas). The same String.Manipulation.cs changes that affected the other Split tests likely impact this path as well. Corroborated by the two sibling Split regressions.
  • Candidates Considered: Same as above
• Verification Status: Not verified

Open Questions

• Historical Data: Split_Csv has zero historical runs; cannot classify signal vs noise. The two Split tests have inflated σ from a regime shift (~26 → ~23.9 ns), making z-scores borderline.
• Confidence Limitations: While the commit attribution is strong (direct code path, exact timestamp match), the statistical metrics are borderline (z barely > 2.0, delta barely exceeds adaptive threshold) due to historical variance artifacts.
• Verification Needed: Run verification with RunWithBuildAtHash at baseline (986fba0) and at e9914e4 to confirm the regression is reproducible and attributable to this specific commit. This would elevate finding confidence to 4/5.
• Bisection Status: Not performed (unnecessary - only 4 commits in range with a clear candidate)

Recommended Actions

1. Verify regression: Run RunWithBuildAtHash comparing 986fba00d9f8 vs e9914e44b440 with the affected Split tests to confirm the char-array path regression on ARM64
2. Source review: Review the String.Manipulation.cs changes in PR #125379 focusing on the char-array overload path - the optimization targets single-char overloads but may have inadvertently regressed the char-array path on ARM64
3. Notify PR author: The PR (Optimize string.Split(char, ...) #125379) by hamarb123 should be notified of this ARM64 regression in the char-array Split path - the PR benchmarks may not have covered this specific variant

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ"...	✅ Culprit found	e9914e44b440	✅ RELATED	23.9→29.0 ns (+21%)	23.7→28.8 ns (+21%)	6462s
Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ"...	✅ Culprit found	e9914e44b440	✅ RELATED	24.0→28.9 ns (+20%)	23.8→29.0 ns (+22%)	2876s
])	🔍 No change found	-	-	73.3→80.7 ns (+10%)	— (not measured)	910s

Total bisection time: 10248s
Success rate: 2/3 tests

System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None)

• Culprit: e9914e44b440
• Commit Message: Optimize string.Split(char, ...) (Optimize string.Split(char, ...) #125379)
• Relevance: RELATED
• Explanation: This commit directly modifies String.Manipulation.cs with 231 additions and 40 deletions optimizing the string.Split(char, ...) code path. The regressing benchmark Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None) calls string.Split(char[], StringSplitOptions) which exercises the exact code modified. The PR's own description notes it focused on "rare/no separators" improvement but may have regressed the char[] overload path for non-matching cases.

Helix jobs

Job ID	Submitted by bisection	Last status
9d05101d-a557-452f-9eac-d61ff961af8e	✅	completed
2a82f959-7aae-4a82-a248-1f0b053eb393	✅	completed
03638352-ba90-4f73-8d66-4a334d652361	✅	completed
3714fe78-31ef-48fe-87b5-f130d4f642ab	✅	completed
6012f436-37c4-4483-ae77-43c3e22e4123	✅	completed
03405a23-0433-464d-b0b6-7e56f59dfbb7	✅	submitted

Per-commit measurements

Commit	Position	Measurements
58dfec78ed8d	baseline, pre-split-optimization	23.747 ns
e9914e44b440	culprit, split optimization pr #125379	28.834 ns

Bisection details

…[earlier output omitted]

Now let me get the full commit summary for the final report:

● get_single_commit_summary (MCP: PerfTriageMcp) · owner: "dotnet", repo: "runtime", sha: "e9914e44…
└ {"index":null,"sha":"e9914e44b44095cfab74d577d438c29024e0cf55","abbrevSha":"e...

The bisection is complete. I have all the data needed for the final report.

BISECTION RESULT:

• Culprit Commit: e9914e4
• Commit Message: Optimize string.Split(char, ...) (Optimize string.Split(char, ...) #125379)
• Commits Tested: 2
• Confidence: 3/5
• Commit Relevance: RELATED
• Relevance Explanation: This commit directly modifies String.Manipulation.cs with 231 additions and 40 deletions optimizing the string.Split(char, ...) code path. The regressing benchmark Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: None) calls string.Split(char[], StringSplitOptions) which exercises the exact code modified. The PR's own description notes it focused on "rare/no separators" improvement but may have regressed the char[] overload path for non-matching cases.
• Baseline Value: 23.747 ns
• Culprit Value: 28.834 ns
• Summary: Bisected a 4-commit range. Testing commit 58dfec7 (before) showed 23.747 ns matching baseline, while commit e9914e4 (the Split optimization) showed 28.834 ns matching the regressed value. Confidence capped at 3/5 due to high noise floor (CV=10.16%, 2σ≈20.3% of baseline) making single-sample measurements less definitive, though the measured delta (~21.4%) exceeds 2σ.

HELIX JOBS:

• 9d05101d-a557-452f-9eac-d61ff961af8e
• 2a82f959-7aae-4a82-a248-1f0b053eb393
• 03638352-ba90-4f73-8d66-4a334d652361
• 3714fe78-31ef-48fe-87b5-f130d4f642ab
• 6012f436-37c4-4483-ae77-43c3e22e4123
• 03405a23-0433-464d-b0b6-7e56f59dfbb7

MEASUREMENTS:

• 58dfec7 (baseline, pre-Split-optimization): 23.747 ns
• e9914e4 (culprit, Split optimization PR Optimize string.Split(char, ...) #125379): 28.834 ns

System.Tests.Perf_String.Split(s: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", arr: [' '], options: RemoveEmptyEntries)

• Culprit: e9914e44b440
• Commit Message: Optimize string.Split(char, ...) (Optimize string.Split(char, ...) #125379)
• Relevance: RELATED
• Explanation: This commit directly modifies String.Manipulation.cs with 231 additions optimizing string.Split(char, ...). The regressed test exercises exactly this code path — splitting a string by a char separator with RemoveEmptyEntries. The optimization improved cases with many separators but regressed the zero-separator-found case tested here.

Helix jobs

Job ID	Submitted by bisection	Last status
6964e602-2fea-4d9c-a0f7-bc82258f6445	✅	completed
4a6bb75a-2970-4c46-9148-6e3ba258e130	✅	completed
d128e939-94c9-400e-aad9-c04954c8e283	✅	completed
83bc141e-dd6c-4568-a2b0-013e26d775bb	✅	submitted

Per-commit measurements

Commit	Position	Measurements
986fba00d9f8	baseline	23.570 ns
58dfec78ed8d	good	23.764 ns
8f365c934088	compare	28.930 ns
e9914e44b440	culprit	28.989 ns

Bisection details

…[earlier output omitted]

4e44b44095cfab74d577d438c29024e0cf55","abbrevSha":"e...

The bisection is complete. The culprit is clearly commit e9914e44b44095cfab74d577d438c29024e0cf55 — titled "Optimize string.Split(char, ...) (#125379)". This commit directly modifies String.Manipulation.cs and SpanHelpers.Packed.cs, which are exactly the code paths exercised by the Perf_String.Split benchmark.

The Helix job confirmed:

• 58dfec78 (previous commit): 23.764 ns (matches baseline)
• e9914e44 (this commit): 28.989 ns (regressed)

The regression delta is ~5.2 ns (+22%), matching the reported +20.2% regression. Despite the PR title saying "Optimize", it appears to have regressed the specific case of splitting a string with no separators found (the "ABCDEFGHIJKLMNOPQRSTUVWXYZ" string split by ' ' has zero matches).

BISECTION RESULT:
- Culprit Commit: e9914e44b44095cfab74d577d438c29024e0cf55
- Commit Message: Optimize `string.Split(char, ...)` (#125379)
- Commits Tested: 5
- Confidence: 5
- Commit Relevance: RELATED
- Relevance Explanation: This commit directly modifies `String.Manipulation.cs` with 231 additions optimizing `string.Split(char, ...)`. The regressed test exercises exactly this code path — splitting a string by a char separator with RemoveEmptyEntries. The optimization improved cases with many separators but regressed the zero-separator-found case tested here.
- Baseline Value: 23.764 ns
- Culprit Value: 28.989 ns
- Summary: Bisected a 4-commit range between 986fba00 and 8f365c93. Helix job confirmed a clear step change at e9914e44, which is PR #125379 optimizing string.Split(char). The "no separator found" case regressed ~22% on ARM64.

HELIX JOBS:
- 6964e602-2fea-4d9c-a0f7-bc82258f6445
- 4a6bb75a-2970-4c46-9148-6e3ba258e130
- d128e939-94c9-400e-aad9-c04954c8e283
- 83bc141e-dd6c-4568-a2b0-013e26d775bb

MEASUREMENTS:
- 986fba00d9f8 (baseline): 23.57 ns
- 58dfec78ed8d (good): 23.764 ns
- e9914e44b440 (culprit): 28.989 ns
- 8f365c934088 (compare): 28.93 ns

System.Tests.Perf_String.Split_Csv(testName: "Long Text", lines: ["Use Case ID/Name,Mission Area,Agency,Bureau/Component/Office,Topic Area,Intended purpose and expected benefits of use case,Description of AI system's outputs,Current stage of development,Rights-impacting or Safety-impacting,Date Initiated,Development/Acquisition Date,Date Implemented,Date Retired,Open-source link to project code (if available),Supporting a High Impact Service Provider (HISP),"If yes to supporting a HISP, which one?",Public-facing service in HISP,Developed under contract(s) or in-house,Agency-owned Data Description,Demographic variables used in model features", "USDA-001: Repair Spend,"REE: Research, Education, and Economics",ARS: Agricultural Research Service,Administrative and Financial Management,Mission-Enabling (Internal Agency Support),"The intended purpose of this model is to review financial documents and then classify each expense as money spent on ""facility repairs"" or ""not facility repairs"". The expected benefits include reduction of manual hours identifying the types of transactions.","The output of the model is a recommendation of which financial transactions should be identified as ""repair"" expenses.","Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,10/1/2019,10/1/2019,6/9/2020,,,No,,,Developed with both contracting and in-house resources,"Approximately 14,000 financial transactions were used to train the model and finetune its parameters. Approximately 3,500 financial transactions were used to test the performance of the final model.",None;", "USDA-002: ARS Project Mapping ,"REE: Research, Education, and Economics",ARS: Agricultural Research Service,Office of National Programs,Science & Space,"The intended purpose of this model is to process research plans from various research program portfolios in the Agricultural Research Service (ARS) to find patterns and opportunities between projects. The expected benefits include decreasing the time that humans would spend to manually read, pull out key terms, and group the projects by topic. The model may also find patterns that a human might miss.","The model outputs groups of similar projects and project terms. The output includes metrics (silhouette scores, term rank, importance scores) that show how well the projects and terms in a group match.","Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,1/1/2020,1/1/2021,5/1/2022,,,No,,,Developed with contracting resources,"The data is a collection of project plans written by scientists, roughly 600 text documents. The texts are related to publicly available 5-year action plans. ",None;", "USDA-003: NAL Automated Indexing,"REE: Research, Education, and Economics",ARS: Agricultural Research Service,National Agricultural Library,Science & Space,This system automatically assigns word tags to agricultural research articles from a controlled list of terms provided by the National Agricultural Library Thesaurus (NALT). The tags can be used to look up and retrieve articles. Using these tags benefits users by making it easier to find the content they are looking for.,The model outputs terms to use as search tags that are specific to the article that the model analyzed.,"Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,6/1/2011,1/1/2012,6/1/2012,,,No,,,Developed with both contracting and in-house resources,The data is a collection of text scripts from publishers that have undergone quality assurance and quality control processing.,None;", "USDA-004: Predictive Modeling of Invasive Pest Species,MRP: Marketing and Regulatory Programs,APHIS: Animal and Plant Health Inspection Service,Plant Protection and Quarantine,Mission-Enabling (Internal Agency Support),"The purpose of the model is to check how likely it is for imported agricultural products from other countries to have pests. Benefits include more reliable discovery and quarantine of invasive pests, preventing pest invasion and making trade safer.",The model outputs are a prediction of whether a product carries an invasive species and what invasive species category the pest belongs to.,"Stage 4 - Operation and Maintenance (Use case is integrated into agency operations, and is being monitored for performance)",Neither,7/1/2015,7/1/2015,5/1/2018,,,No,,,Developed in-house,Inspection data was collected from Plant Protection and Quarantine (PPQ) and Customs and Border Protection (CBP). Quality control of the data was conducted by data analysts. There was no data augmention performed. The data is structured and has several million records with more than 50 numerical and categorical variables. This data is not publicly available. ,None;", ...])

• Status: inconclusive

Helix jobs

Job ID	Submitted by bisection	Last status
c1e20b06-ce00-4169-a357-f127b2b6c6be	✅	submitted

Bisection details

…[earlier output omitted]

└ {"jobId":"c1e20b06-ce00-4169-a357-f127b2b6c6be","status":"running","message":...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "c1e20b06-ce00-4169-a357-f127b2b6c6be"
└ {"jobId":"c1e20b06-ce00-4169-a357-f127b2b6c6be","status":"running","message":...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "c1e20b06-ce00-4169-a357-f127b2b6c6be"
└ {"jobId":"c1e20b06-ce00-4169-a357-f127b2b6c6be","status":"running","message":...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "c1e20b06-ce00-4169-a357-f127b2b6c6be"
└ {"jobId":"c1e20b06-ce00-4169-a357-f127b2b6c6be","status":"running","message":...

● get_helix_job_result (MCP: PerfTriageMcp) · jobId: "c1e20b06-ce00-4169-a357-f127b2b6c6be"
└ {"jobId":"c1e20b06-ce00-4169-a357-f127b2b6c6be","status":"running","message":...

I've been polling the Helix job for an extended period (well over 120+ polling attempts spanning the full session duration). The ARM64 queue (AzureLinux.3.Cobalt.Arm64.Perf) has not picked up the work items despite the job being in "running" status with 2 work items found. This exceeds my polling budget.

BISECTION FAILED:
- Reason: Helix job exceeded polling budget — ARM64 queue never completed work items after 120+ minutes of polling
- Helix Errors: No error from Helix itself; job c1e20b06-ce00-4169-a357-f127b2b6c6be remained in "running" status with "Found 2 work items" indefinitely on queue AzureLinux.3.Cobalt.Arm64.Perf
- What was attempted: Submitted SubmitHelixJob with all 5 commits (baseline + 4 in range) to AzureLinux.3.Cobalt.Arm64.Perf queue with arm64/linux/net11.0/Release config. Polled GetHelixJobResult over 120 times across the full session. Job stayed "running" with 2 work items found but never completed.
- What is needed: Retry when the ARM64 Cobalt Perf queue has available capacity, or investigate whether the queue AzureLinux.3.Cobalt.Arm64.Perf is offline/backlogged

🤖 Proposed Next Actions

Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

Verified regression attributed to commit e9914e4 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

#	Action	Status	Details
1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for e9914e44b440
2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
3	Ping owners	⏭️ Skipped (dry-run)	no owners resolved
4	Remove label	⏭️ Skipped (dry-run)	untriaged

🦆 Pre-bisection Re-analysis (advisory)

✅ Agrees with the pre-bisection plan — advisory only, not applied

Rationale: All three findings share the same likely commit (e9914e4) with relevant stack traces and no bimodal history, which mutually reinforces the signal. The Split_Csv test has weaker confidence (2/5 finding, 3/5 change) but since it targets the same code path and commit, including it provides useful corroboration without wasting a bisection slot on a divergent hypothesis.

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

⚠️ Disputes the analysis — advisory only, not applied

Flags: overconfident, weak-per-finding-evidence

Rationale: The overall 5/5 confidence scores are not justified when the per-finding signals max out at 3/5 finding confidence and 4/5 change confidence, and the Split_Csv bisection was inconclusive. Two successful bisections converging on the same SHA support a 4/5 change confidence, but not a perfect 5/5.

Suggested confidence: Finding Confidence → 4/5; Change Confidence → 4/5 (advisory — not applied)

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-07-09 13:23 UTC

[DrewScoggins]

Also tagging @hamarb123