Flaky-test: fix duplicate-issue bug by switching dedup key to a visible code-block key - #13963
Merged
Conversation
…de-block key gh-aw's safe-output sanitizer strips agent-authored HTML comments, so the hidden <!-- flaky-test-id --> marker never survived in filed issue/PR bodies, causing the same test to be re-filed on consecutive daily runs. Switch to a visible 'flaky-test-id: <testName>' key rendered in a fenced code block (survives sanitization and matches what Get-FlakyTests.ps1 already searches for). Dedup now matches the key as a complete line (not a substring, avoiding MyTest vs MyTestExtended collisions) in the body OR the normalized FQN in the title, in either issues or open PRs. Scope the issue title fallback to flaky-test-labeled issues. Drop two unread hidden status markers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the flaky-test detector/fixer guidance to stop using hidden HTML-comment markers for cross-run de-duplication (which are stripped by gh-aw sanitization) and instead use a visible flaky-test-id: <testName> token rendered in a fenced code block.
Changes:
- Update the detector and fixer agent instructions to emit/search a visible
flaky-test-id: <testName>key (and remove references to hidden markers). - Refine de-dup guidance to avoid substring collisions by requiring whole-line body matches (and allow title-based matching as a fallback).
- Update
Get-FlakyTests.ps1commentary to refer to the visible key.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/scripts/Get-FlakyTests.ps1 | Updates Step 4 comment to reference the new visible dedup key. |
| .github/workflows/flaky-test-fixer.agent.md | Replaces hidden marker dedup instructions with visible flaky-test-id key guidance for PR bodies. |
| .github/workflows/flaky-test-detector.agent.md | Replaces hidden marker issue/PR dedup instructions with visible key + title fallback guidance, and scopes title search by label in the instructions. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 1
… literal An unquoted 'flaky-test-id: <FQN>' GitHub search makes GitHub parse 'flaky-test-id:' as a (nonexistent) qualifier and drop it, degrading the candidate search to just the test name. Wrap the value in double quotes so it is matched as a literal phrase, matching the quoted form the agent instructions already use. (PR #13963 review feedback.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two fixer runs reached opposite diagnoses on the same 100%-consistently-failing test (#11498): one tried a test-assertion patch, the other correctly noop'd. The issue body documents it as a deterministic product regression from PR #11474 (the maintainer even says the new behavior is 'probably ok'), but the fixer only read 'gh issue view --json state' and never saw that, so the LLM re-derived the root cause non-deterministically. Add two guardrails: (1) read the tracking issue body + recent comments and defer (leave quarantined) when a maintainer attributes the failure to a product change / specific PR / calls it by-design; (2) treat a near-100%-consistent break with no test-side nondeterminism story as a likely regression and skip, rather than papering over it with a test change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ViktorHofer
enabled auto-merge (squash)
June 5, 2026 15:16
JanKrivanek
approved these changes
Jun 5, 2026
This was referenced Jun 6, 2026
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The flaky-test detector and auto-fixer used a hidden HTML comment marker (
<!-- flaky-test-id: <FQN> -->) in filed issue/PR bodies as the cross-run de-duplication key. gh-aw's safe-output sanitizer strips agent-authored HTML comments, so the marker never survived in the published body. The dedup search therefore always came up empty and the same flaky test was re-filed as a brand-new issue on consecutive daily runs (e.g. #13922 / #13948 / #13949 / #13950 / #13951 were duplicates of the canonical Jun-3 issues).This was proven empirically: the bot-authored
flaky-test-idcomment is absent from every filed body, while gh-aw's owngh-aw-workflow-idfooter comment survives (it is appended after sanitization).Fix
Switch the dedup key from a hidden comment to a visible
flaky-test-id: <testName>token, rendered in a fenced code block in every filed issue body and every per-test PR section. A visible token survives sanitization, and it matches whatGet-FlakyTests.ps1already searches for (--search "flaky-test-id: <name> in:body,title"), so no scan-engine logic change is required.De-dup rule at every site (issue filing, in-flight-PR checks, fixer selection, pre-PR re-check):
flaky-test-id: <testName>as a complete line (not a substring — avoidsMyTestspuriously matchingMyTestExtended) in the body, or<testName>in the title,Also:
flaky-test-labeled issues, so an unrelated issue that merely mentions a method name can't suppress a real new flake.flaky-test-fix/flaky-test-unquarantine) that nothing ever read.Scope
Instruction-only (
.mdbodies are runtime-imported) plus a one-line comment inGet-FlakyTests.ps1— no frontmatter change, so no lock recompile. The duplicate issues this addresses were already closed manually.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com