[ci-scan-feedback] Strengthen occurrence thresholds, dedup search, and do-not-disable detection

[github-actions]

Triggering signals

• (PR [ci-scan] Skip stackoverflowtester under interpreter mode (refs #127899) #128737, @janvorli: "I don't think we should disable the test without trying to understand the problem", [ci-scan] Skip stackoverflowtester under interpreter mode (refs #127899) #128737 (comment))
• (PR [ci-scan] Skip stackoverflowtester under interpreter mode (refs #127899) #128737, @BrzVlad: "I'm also very aggressive with fixing these pipelines without disabling any tests", [ci-scan] Skip stackoverflowtester under interpreter mode (refs #127899) #128737 (comment))
• (PR [ci-scan] Skip HostTests Moq tests on iossimulator (refs #128405) #128469, @svick: "Fix the compile error: Argument 3: cannot convert from 'Xunit.TestRuntimes' to 'Xunit.TargetFrameworkMonikers'", [ci-scan] Skip HostTests Moq tests on iossimulator (refs #128405) #128469 (comment))
• ([ci-scan] Build break: Crossgen2 OOM on HugeArray1.dll (windows x86 R2R_CG2) #128697, closed not_planned: duplicate of [ci-scan] Build break: Crossgen2 OOM compiling HugeArray1.dll on x86 R2R #128531 — shared KBE search missed existing issue due to ErrorMessage form mismatch)
• ([ci-scan] Test failure: X509Certificate2CollectionRemoveRangeArray under crossgen2 R2R #128538, [ci-scan] Test failure: paltest_pal_sxs_test1 on osx-arm64 #128530, [ci-scan] Test failure: Avx512F.FusedMultiplyAddNegated assertion in jitstress2-jitstressregs #128544: closed by maintainer as "not seen since creation" — one-off failures filed prematurely)
• (PR [ci-scan] Skip X509Certificate2CollectionRemoveRangeArray under R2R (refs #128538) #128703: references closed KBE [ci-scan] Test failure: X509Certificate2CollectionRemoveRangeArray under crossgen2 R2R #128538, now orphaned)

Proposed edits

• (.github/workflows/ci-failure-scan.md:262, strengthen Step 4.7 to read referenced issues and expand do-not-disable keyword list — tied to PR [ci-scan] Skip stackoverflowtester under interpreter mode (refs #127899) #128737)
• (.github/workflows/ci-failure-scan.md:244-258, add variant-form dedup guidance for KBE search — tied to [ci-scan] Build break: Crossgen2 OOM on HugeArray1.dll (windows x86 R2R_CG2) #128697)
• (.github/workflows/ci-failure-scan.md:290, clarify 2-occurrence threshold must span distinct builds — tied to [ci-scan] Test failure: X509Certificate2CollectionRemoveRangeArray under crossgen2 R2R #128538/[ci-scan] Test failure: paltest_pal_sxs_test1 on osx-arm64 #128530)
• (.github/workflows/ci-failure-scan.md:296, verify linked KBE still open before Branch B emit — tied to PR [ci-scan] Skip X509Certificate2CollectionRemoveRangeArray under R2R (refs #128538) #128703)
• (.github/workflows/ci-failure-scan.md:389, document full ActiveIssue 4-param constructor — tied to PR [ci-scan] Skip HostTests Moq tests on iossimulator (refs #128405) #128469)

Expected behavior change

The next run will (1) avoid filing KBEs for one-off failures that appear in only a single build, (2) detect existing KBEs even when filed with a different signature representation (array vs literal), (3) catch do-not-disable signals on root-cause issues referenced by the KBE, (4) not emit orphaned test-disable PRs against closed KBEs, and (5) use the correct ActiveIssue constructor overload to avoid compile errors.

Generated by CI Outer-Loop Failure Scanner — Feedback · ● 14.1M · ◷

---

Note

This was originally intended as a pull request, but the git push operation failed.

Workflow Run: View run details and download patch artifact

The patch file is available in the agent artifact in the workflow run linked above.

To create a pull request with the changes:

# Download the artifact from the workflow run
gh run download 26675071020 -n agent -D /tmp/agent-26675071020

# Create a new branch
git checkout -b ci-scan-feedback/strengthen-thresholds-and-dedup-20260530-52c7124b3554fbda

# Apply the patch (--3way handles cross-repo patches where files may already exist)
git am --3way /tmp/agent-26675071020/aw-ci-scan-feedback-strengthen-thresholds-and-dedup-20260530.patch

# Push the branch to origin
git push origin ci-scan-feedback/strengthen-thresholds-and-dedup-20260530-52c7124b3554fbda

# Create the pull request
gh pr create --title '[ci-scan-feedback] Strengthen occurrence thresholds, dedup search, and do-not-disable detection' --base main --head ci-scan-feedback/strengthen-thresholds-and-dedup-20260530-52c7124b3554fbda --repo dotnet/runtime

Show patch preview (73 of 73 lines)

From 1b1351570f657b8e5edcc02ef127ac23a860459b Mon Sep 17 00:00:00 2001
From: "github-actions[bot]" <github-actions[bot]@users.noreply.github.com>
Date: Sat, 30 May 2026 05:03:16 +0000
Subject: [PATCH] [ci-scan-feedback] Strengthen thresholds, dedup search, and
 do-not-disable detection

- Step 4.7: Read referenced issues for do-not-disable signals; expand
  keyword list with phrases seen in maintainer rejections (PR #128737)
- Step 4.2-4.6: Add guidance to search variant signature forms (array
  vs literal ErrorMessage) to prevent duplicate KBEs (#128697)
- Branch A: Clarify that 2-occurrence threshold must span distinct
  builds, not just distinct work items in one build (#128538, #128530)
- Branch B: Verify linked KBE is still open before emitting PR (#128703)
- Document full ActiveIssue 4-parameter constructor overload (#128469)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---
 .github/workflows/ci-failure-scan.md | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/ci-failure-scan.md b/.github/workflows/ci-failure-scan.md
index 02270d6b..5fcf0315 100644
--- a/.github/workflows/ci-failure-scan.md
+++ b/.github/workflows/ci-failure-scan.md
@@ -250,6 +250,8 @@ Follow exactly these sections from `.github/workflows/shared/create-kbe.instruct
 3. `<a id="search-existing-prs"></a>` / `## Search for existing PRs already handling the failure`
 4. `<a id="verify-embedded-issues"></a>` / `## Verify every embedded issue number exists`
 
+When searching for existing KBEs, account for signature form variants: the same error may appear as a literal `ErrorMessage` string in one KBE and as an `ErrorMessage` JSON array in another. Search using both the full signature AND key substrings (test name, exception type, assertion file:line) to catch duplicates filed with different signature representations.
+
 Record the same outcomes described there:
 
 - `existing-kbe #<n>`
@@ -259,9 +261,9 @@ Record the same outcom
... (truncated)