Skip to content

Harden test-quarantine workflow guardrails (B1/B2/B3)#67764

Merged
wtgodbe merged 2 commits into
dotnet:mainfrom
wtgodbe:wtgodbe/quarantine-guardrail-fixes
Jul 13, 2026
Merged

Harden test-quarantine workflow guardrails (B1/B2/B3)#67764
wtgodbe merged 2 commits into
dotnet:mainfrom
wtgodbe:wtgodbe/quarantine-guardrail-fixes

Conversation

@wtgodbe

@wtgodbe wtgodbe commented Jul 13, 2026

Copy link
Copy Markdown
Member

Fixes three bad behaviors observed in recent test-quarantine runs. All changes are to the workflow spec (test-quarantine.md) and its compiled test-quarantine.lock.yml; both were reviewed to consensus by two code-review agents on different models.

B1 — Stray commits leaking across sibling PRs

A change from one candidate could leak into another candidate's PR branch (e.g. #67753 carried an unrelated WebSocketsWorks re-quarantine + revert).

Adds Step 3.0 "Branch hygiene": every PR must start from a clean main — clean the working tree/index first, then git checkout -f main (documented that git checkout main before cleanup aborts on a dirty tree and would silently strand the agent on the prior branch). Adds a pre-submit git log main..HEAD / git diff main...HEAD self-check so committed stray changes are caught, not just working-tree ones.

B2 — Re-quarantining a test a maintainer already rejected

Run 29245029267 re-quarantined WebSocketsWorks despite a maintainer having closed a prior attempt (#67443) asking it not to retry without fresh evidence.

The closed_quarantine_prs pre-activation feed now emits closed_at, closed_by, and a precomputed trusted_closed. The Important-Rules and Case B logic now impose a date cutoff instead of trying to interpret a conditional English comment:

  • Quarantine / re-quarantine: count only failures whose build startedUtc is strictly after the latest cutoff-establishing close; defer if too few remain.
  • Unquarantine (passing-streak driven, so a failure cutoff is meaningless): blanket-skip for the run when a cutoff-establishing close exists.

trusted_closed is derived only from token-safe signals: a non-author human closing this bot-authored PR (which necessarily holds triage/write permission — covers public and private org members with zero extra API calls), public dotnet org membership, or a trusted-author comment. No fix grows a pre-activation output past the per-env-var limit; the feed still self-caps and fails closed.

B3 — Unquarantining a permanently re-quarantined test

Run 29149418882 opened #67740 to unquarantine a test whose issue (#66119) is permanently re-quarantined.

Adds a mandatory pre-PR diff self-check before every unquarantine: for every removed QuarantinedTest attribute referencing an issue URL — method-, class-, or [assembly: ...]-level — extract the issue number and abort if it appears in requarantine_issue_numbers. Fails closed if that list is missing.


Nothing merges an unrelated file; only the two workflow files change. gh aw compile is clean (0 errors/warnings) and the pinned action SHAs are unchanged.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Fixes three bad behaviors observed in recent test-quarantine runs:

- B1: stray commits from one candidate leaking into a sibling PR's branch.
  Adds Step 3.0 "Branch hygiene" requiring each PR to start from a clean
  main (clean the tree, then git checkout -f main) and a pre-submit
  git log main..HEAD / git diff main...HEAD self-check.

- B2: re-quarantining a test whose prior attempt a maintainer already
  rejected by closing the PR. The closed_quarantine_prs pre-activation feed
  now emits closed_at/closed_by/trusted_closed; the Important-Rules and
  Case B logic impose a date cutoff (count only failures after a trusted
  close for quarantine/re-quarantine; blanket-skip for unquarantine).
  trusted_closed is derived from token-safe signals (non-author human
  closer of a bot PR, public dotnet org membership, or a trusted comment).

- B3: unquarantining a permanently re-quarantined test. Adds a mandatory
  pre-PR diff self-check that aborts if any removed QuarantinedTest
  attribute (method-, class-, or assembly-level) references an issue in
  requarantine_issue_numbers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c6695e66-ef29-4591-8ca6-fc494164b1ca
Copilot AI review requested due to automatic review settings July 13, 2026 18:28
@wtgodbe wtgodbe requested a review from a team as a code owner July 13, 2026 18:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the test-quarantine agentic workflow guardrails to prevent (1) cross-PR branch contamination, (2) re-proposing quarantines/unquarantines after a trusted maintainer has already closed a prior attempt, and (3) unquarantining tests that are permanently barred due to a prior re-quarantine.

Changes:

  • Extend the “closed test-quarantine PRs” pre-activation feed to include closed_at, closed_by, and a token-safe trusted_closed signal for applying a cutoff based on trusted closes.
  • Update the workflow guidance to apply a “failures after cutoff only” rule for quarantine/re-quarantine, and to skip unquarantines entirely when a cutoff-establishing prior close exists.
  • Add explicit “Branch hygiene” and a mandatory pre-PR diff gate to prevent unquarantining tests whose issue is in requarantine_issue_numbers.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/test-quarantine.md Updates workflow spec: richer closed-PR metadata/trust signals, cutoff-based rules, branch-hygiene instructions, and a mandatory unquarantine re-quarantine diff gate.
.github/workflows/test-quarantine.lock.yml Regenerated compiled workflow reflecting the updated spec (no action SHA changes).

Comment on lines +1522 to +1526
```
git checkout -- .
git reset --hard HEAD
git checkout -f main
```

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — applied in 628b604. Added git clean -fd to the branch-hygiene sequence (before switching back to main) so untracked files/directories left by a prior candidate can't leak into the next PR. I intentionally omitted -x so gitignored build artifacts are preserved.

…ates

Addresses PR review feedback: reset/checkout leave untracked files in
place, so a stray file created by a prior candidate could be added into
the next PR. Add `git clean -fd` before switching back to main; omit -x
so gitignored build artifacts are preserved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c6695e66-ef29-4591-8ca6-fc494164b1ca
@wtgodbe wtgodbe merged commit 56722f8 into dotnet:main Jul 13, 2026
10 checks passed
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 11.0-preview7 milestone Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants