[Flaky Test] Un-quarantine 5 consistently-green tests - #13952
Conversation
Remove [ActiveIssue] from tests that have been passing consistently in the quarantine pipeline (definition 344) over 4+ builds / 3+ days across all applicable platforms: - BuildManager_Tests.SimpleGraphBuild (issue #4368, all 3 platforms) - BuildManager_Tests.GraphBuildValid (issue #4368, all 3 platforms) - BuildManager_Tests.GraphBuildFail (issue #4368, all 3 platforms) - BuildEnvironmentHelper_Tests.BuildEnvironmentFindsAmd64RunningInAmd64NoVS (issue #7552, Windows) - BuildEnvironmentHelper_Tests.BuildEnvironmentFindsAmd64NoVS (issue #7552, Windows) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
the
flaky-test
There was a problem hiding this comment.
Pull request overview
This PR un-quarantines five previously flaky unit tests by removing their [ActiveIssue] attributes, restoring them to regular execution in the normal test pipelines while leaving other still-flaky tests quarantined.
Changes:
- Removed
[ActiveIssue(".../issues/7552")]from twoBuildEnvironmentHelper_TestsWindows-only test cases. - Removed
[ActiveIssue(".../issues/4368")]from threeBuildManager_Testsstatic-graph test cases. - Kept other
[ActiveIssue]quarantines intact (e.g.,GraphBuildInvalid,GraphBuildCircular, andBuildEnvironmentFindsAmd64RunningInAmd64still reference the same issues).
Show a summary per file
| File | Description |
|---|---|
| src/Build.UnitTests/BuildEnvironmentHelper_Tests.cs | Un-quarantines two Windows-only BuildEnvironmentHelper tests by removing the [ActiveIssue] attribute. |
| src/Build.UnitTests/BackEnd/BuildManager_Tests.cs | Un-quarantines three static graph BuildManager tests by removing the [ActiveIssue] attribute while leaving other quarantined graph tests unchanged. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 0
…es section from PR body (#13958) Follow-up to #13938, addressing two pieces of feedback on how the flaky-test **detector** opens its combined quarantine/un-quarantine PR. ### 1. Open quarantine/un-quarantine PRs ready-for-review (not draft) The detector previously opened its PR as a draft (`draft: true`). dnceng AzDO build validation does not run on draft PRs, so CI only kicked in once a maintainer marked the PR ready for review (observed on #13952). Since these PRs are mechanical `[ActiveIssue]` add/removes that benefit from immediate CI validation, switch to `draft: false` so CI runs as soon as the PR is opened. ### 2. Don't list newly-filed tracking issues in the PR body On an un-quarantine PR (#13952), the agent appended a "New flaky test issues filed this run" section referencing freshly-filed issues (e.g. #13949-#13951). Those issues are **not** acted on by the PR (they only become quarantine-eligible on a later run), so the `#number` references created a misleading issue<->PR cross-link. Step 8 now explicitly instructs the agent to reference only the issues for tests it actually quarantines or un-quarantines in that PR. Frontmatter changed (`draft` + `description`), so the lock file was regenerated; cron is preserved at `38 11`. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Un-quarantines 5 tests that have been passing consistently in the quarantine pipeline (AzDO definition 344) over ≥ 4 distinct builds / ≥ 3 distinct days across all applicable platforms.
Un-quarantined tests
BuildManager_Tests.SimpleGraphBuild— Un-quarantine (7e)Tracked by #4368
Def-344 evidence: 4 distinct builds, 3 distinct days, legs: Quarantine (Windows), Quarantine test logs (Linux), Quarantine test logs (macOS). Not present in def-344
flakyTestsover the 21-day window.BuildManager_Tests.GraphBuildValid— Un-quarantine (7e)Tracked by #4368
Def-344 evidence: 4 distinct builds, 3 distinct days, legs: Quarantine (Windows), Quarantine test logs (Linux), Quarantine test logs (macOS). Not present in def-344
flakyTestsover the 21-day window.BuildManager_Tests.GraphBuildFail— Un-quarantine (7e)Tracked by #4368
Def-344 evidence: 4 distinct builds, 3 distinct days, legs: Quarantine (Windows), Quarantine test logs (Linux), Quarantine test logs (macOS). Not present in def-344
flakyTestsover the 21-day window.BuildEnvironmentHelper_Tests.BuildEnvironmentFindsAmd64RunningInAmd64NoVS— Un-quarantine (7e)Tracked by #7552
Def-344 evidence: 4 distinct builds, 3 distinct days, leg: Quarantine (Windows). This is a
[WindowsOnlyFact], so Windows-only green evidence is sufficient coverage.BuildEnvironmentHelper_Tests.BuildEnvironmentFindsAmd64NoVS— Un-quarantine (7e)Tracked by #7552
Def-344 evidence: 4 distinct builds, 3 distinct days, leg: Quarantine (Windows). This is a
[WindowsOnlyFact], so Windows-only green evidence is sufficient coverage.New flaky test issues filed this run
The following tests were newly detected as flaky but have no pre-existing tracking issue. Issues were filed; they become quarantine-eligible on the next run.
ToolTask_Tests.ToolTaskCanChangeCanonicalErrorFormat(6 sources, Windows)MSBuildTaskHostTests.CompileNet35WinFormsApp(3 sources, Windows timeout)ToolTask_Tests.OverrideStdOutImportanceToHigh(3 sources, Windows)BinaryLoggerTests.BinaryLoggerShouldEmbedFilesViaTaskOutput(3 sources, Linux)