Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion flaky-tests/detection/pass-on-retry-monitor.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: Detect tests that fail then pass on retry within the same commit

The pass-on-retry monitor detects the most common flakiness pattern: a test fails, is retried, and passes on the same commit. This indicates the failure wasn't caused by a code change and that the test is unreliable.

This monitor is branch-agnostic. It evaluates all test runs regardless of which branch they ran on.
By default, this monitor evaluates test runs on all branches. You can scope it to specific branches to focus detection where pass-on-retry behavior is actually meaningful.

## How It Works

Expand Down Expand Up @@ -36,11 +36,26 @@ default 7-day recovery period visible. -->
|---|---|---|
| **Enabled** | Whether the monitor is active | On |
| **Recovery days** | Days without pass-on-retry behavior before a test is resolved as healthy. Range: 1 to 15 days. | 7 |
| **Branch scope** | Which branches the monitor evaluates. Accepts branch names and glob patterns. | All branches (`*`) |

### What Recovery Days Controls

A shorter recovery period (e.g., 1 to 3 days) returns tests to healthy quickly, which is useful if you fix flaky tests promptly and want fast feedback. A longer recovery period (e.g., 10 to 15 days) is more conservative. It keeps tests flagged longer to account for flaky behavior that only surfaces occasionally.

### Branch Scope

Use the **Branch scope** setting to restrict the monitor to a specific set of branches.

This is useful when PR branches generate too much noise. CI often retries tests on pull request branches automatically; if those retries aren't meaningful signals for your team, you can limit detection to stable branches like `main`.

Branch scope uses the same glob syntax as [failure rate monitor branch patterns](failure-rate-monitor.md#branch-pattern-syntax) and accepts up to 10 patterns. Type a pattern and press **Enter** or comma to add it as a chip. For example:

- `main`: only stable branch runs
- `main` and `release/*`: stable plus release branches
- `*` (default): all branches

Changes to branch scope take effect for newly detected events. Previously detected flaky tests are not re-evaluated.

## When Detection Happens

Pass-on-retry detection runs continuously as new test results arrive. A failure and its corresponding retry don't need to arrive at exactly the same time.
Expand Down
Loading