From c57aa18cd4c0500524dbb2ced7652bdd655062f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 7 May 2026 10:56:46 +0000 Subject: [PATCH 1/3] docs(flaky-tests): document branch scope configuration for pass-on-retry monitor trunk2 PRs #3843 and #3909 added backend and UI support for restricting pass-on-retry detection to configured branches. Documents the new Branch scope setting and explains when scoping to stable branches is useful. Co-Authored-By: Claude Opus 4.6 --- flaky-tests/detection/pass-on-retry-monitor.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/flaky-tests/detection/pass-on-retry-monitor.md b/flaky-tests/detection/pass-on-retry-monitor.md index 93bc17ff..5581be1a 100644 --- a/flaky-tests/detection/pass-on-retry-monitor.md +++ b/flaky-tests/detection/pass-on-retry-monitor.md @@ -6,7 +6,7 @@ description: Detect tests that fail then pass on retry within the same commit The pass-on-retry monitor detects the most common flakiness pattern: a test fails, is retried, and passes on the same commit. This indicates the failure wasn't caused by a code change and that the test is unreliable. -This monitor is branch-agnostic. It evaluates all test runs regardless of which branch they ran on. +By default, this monitor evaluates test runs on all branches. You can scope it to specific branches to focus detection where pass-on-retry behavior is actually meaningful. ## How It Works @@ -36,11 +36,26 @@ default 7-day recovery period visible. --> |---|---|---| | **Enabled** | Whether the monitor is active | On | | **Recovery days** | Days without pass-on-retry behavior before a test is resolved as healthy. Range: 1 to 15 days. | 7 | +| **Branch scope** | Which branches the monitor evaluates. Accepts branch names and glob patterns. | All branches (`*`) | ### What Recovery Days Controls A shorter recovery period (e.g., 1 to 3 days) returns tests to healthy quickly, which is useful if you fix flaky tests promptly and want fast feedback. A longer recovery period (e.g., 10 to 15 days) is more conservative. It keeps tests flagged longer to account for flaky behavior that only surfaces occasionally. +### Branch Scope + +By default the pass-on-retry monitor considers test runs from every branch. You can restrict it to a specific set of branches using the **Branch scope** setting in the monitor config. + +This is useful when PR branches generate too much noise. CI often retries tests on pull request branches automatically; if those retries aren't meaningful signals for your team, you can limit detection to stable branches like `main`. + +Branch patterns use the same glob syntax as the [failure rate monitor](failure-rate-monitor.md#branch-pattern-syntax). For example: + +- `main` — only stable branch runs +- `main, release/*` — stable plus release branches +- `*` (default) — all branches + +Changes to branch scope take effect for newly detected events. Previously detected flaky tests are not re-evaluated. + ## When Detection Happens Pass-on-retry detection runs continuously as new test results arrive. A failure and its corresponding retry don't need to arrive at exactly the same time. From 5dafc361c285e187b08b70a58648951f30989be9 Mon Sep 17 00:00:00 2001 From: Sam Gutentag <1404219+samgutentag@users.noreply.github.com> Date: Thu, 7 May 2026 10:41:12 -0700 Subject: [PATCH 2/3] docs(pass-on-retry-monitor): apply Claude review nits on Branch Scope section Four small fixes flagged by the Claude review on the same-day open of PR #625, plus a UI fact-check against the eng PRs that confirmed Claude's chip-style hypothesis. * Drop the redundant "By default the pass-on-retry monitor considers test runs from every branch" opener at line 47. The intro at line 9 already establishes the same default; the section opener now leads with "Use the **Branch scope** setting to restrict the monitor..." * Document the chip-style input. The eng PR #3909 uses a `TagInput` component (string[] of patterns, max 10, comma-key triggers chip creation), not a comma-joined string field. Replaced the misleading `main, release/*` example with `main` and `release/*` (prose form, matching failure-rate-monitor.md:104) and added a sentence describing the chip add gesture. * Surface the 10-pattern limit inline. The cap is enforced in the UI (`max = 10` in the TagInput component); readers should know. * Cross-reference the failure-rate doc with a slightly tighter phrase ("failure rate monitor branch patterns") since the syntax source of truth lives there. Co-Authored-By: Claude Opus 4.7 (1M context) --- flaky-tests/detection/pass-on-retry-monitor.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flaky-tests/detection/pass-on-retry-monitor.md b/flaky-tests/detection/pass-on-retry-monitor.md index 5581be1a..bb5800c3 100644 --- a/flaky-tests/detection/pass-on-retry-monitor.md +++ b/flaky-tests/detection/pass-on-retry-monitor.md @@ -44,14 +44,14 @@ A shorter recovery period (e.g., 1 to 3 days) returns tests to healthy quickly, ### Branch Scope -By default the pass-on-retry monitor considers test runs from every branch. You can restrict it to a specific set of branches using the **Branch scope** setting in the monitor config. +Use the **Branch scope** setting to restrict the monitor to a specific set of branches. This is useful when PR branches generate too much noise. CI often retries tests on pull request branches automatically; if those retries aren't meaningful signals for your team, you can limit detection to stable branches like `main`. -Branch patterns use the same glob syntax as the [failure rate monitor](failure-rate-monitor.md#branch-pattern-syntax). For example: +Branch scope uses the same glob syntax as [failure rate monitor branch patterns](failure-rate-monitor.md#branch-pattern-syntax) and accepts up to 10 patterns. Type a pattern and press **Enter** or **,** to add it as a chip. For example: - `main` — only stable branch runs -- `main, release/*` — stable plus release branches +- `main` and `release/*` — stable plus release branches - `*` (default) — all branches Changes to branch scope take effect for newly detected events. Previously detected flaky tests are not re-evaluated. From 14fffdc77f475c153d9966ee81341a572504d525 Mon Sep 17 00:00:00 2001 From: Sam Gutentag <1404219+samgutentag@users.noreply.github.com> Date: Thu, 7 May 2026 11:47:36 -0700 Subject: [PATCH 3/3] docs(pass-on-retry-monitor): replace em dashes and clarify chip-input keystroke - Replace em dashes in the Branch Scope example bullets with colons - Reword the chip-add hint from bold comma glyph to "or comma" for readability --- flaky-tests/detection/pass-on-retry-monitor.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/flaky-tests/detection/pass-on-retry-monitor.md b/flaky-tests/detection/pass-on-retry-monitor.md index bb5800c3..191ee070 100644 --- a/flaky-tests/detection/pass-on-retry-monitor.md +++ b/flaky-tests/detection/pass-on-retry-monitor.md @@ -48,11 +48,11 @@ Use the **Branch scope** setting to restrict the monitor to a specific set of br This is useful when PR branches generate too much noise. CI often retries tests on pull request branches automatically; if those retries aren't meaningful signals for your team, you can limit detection to stable branches like `main`. -Branch scope uses the same glob syntax as [failure rate monitor branch patterns](failure-rate-monitor.md#branch-pattern-syntax) and accepts up to 10 patterns. Type a pattern and press **Enter** or **,** to add it as a chip. For example: +Branch scope uses the same glob syntax as [failure rate monitor branch patterns](failure-rate-monitor.md#branch-pattern-syntax) and accepts up to 10 patterns. Type a pattern and press **Enter** or comma to add it as a chip. For example: -- `main` — only stable branch runs -- `main` and `release/*` — stable plus release branches -- `*` (default) — all branches +- `main`: only stable branch runs +- `main` and `release/*`: stable plus release branches +- `*` (default): all branches Changes to branch scope take effect for newly detected events. Previously detected flaky tests are not re-evaluated.