Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion hugo/content/en/tests/flaky_management/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Use the state drop-down to change how a flaky test is handled in your CI pipelin
| ----------- | ----------- |
| {{< ui >}}Active{{< /ui >}} | The test is known to be flaky and is running in CI. |
| {{< ui >}}Quarantined{{< /ui >}} | Keep the test running in the background, but failures don't affect CI status or break pipelines. This is useful for isolating flaky tests without blocking merges. Datadog tags test run events with `@test.test_management.is_quarantined:true` when quarantined. |
| {{< ui >}}Disabled{{< /ui >}} | Skip the test entirely in CI. Use this when a test is no longer relevant or needs to be temporarily removed from the pipeline. Datadog tags test run events with `@test.test_management.is_disabled:true` when disabled. |
| {{< ui >}}Disabled{{< /ui >}} | Skip the test entirely in CI. Use this when a test is no longer relevant or needs to be temporarily removed from the pipeline. Datadog tags test run events with `@test.test_management.is_disabled:true` when disabled. Disabled tests are excluded from the automatic move to {{< ui >}}Fixed{{< /ui >}}. |
| {{< ui >}}Fixed{{< /ui >}} | The test has passed consistently and is no longer flaky. If supported, use the [remediation flow](#confirm-fixes-for-flaky-tests) to confirm the fix and automatically apply this state after it is merged into the default branch. |

<div class="alert alert-info">State actions have minimum version requirements for each programming language's instrumentation library. See <a href="#compatibility">Compatibility</a> for details.</div>
Expand Down Expand Up @@ -88,6 +88,7 @@ Configure automated Flaky Test Policies to govern how flaky tests are handled in
<td>{{< ui >}}Fixed{{< /ui >}}</td>
<td>
<p>If a flaky test no longer flakes for 30 days, it is automatically moved to the Fixed state. This automation is default behavior and can't be customized.</p>
<p>Tests in the {{< ui >}}Disabled{{< /ui >}} state are excluded from this automation and are never automatically moved to {{< ui >}}Fixed{{< /ui >}}. Because disabled tests are skipped in CI, Datadog has no signal about whether they are still flaky. To make a disabled test eligible again, change its state to {{< ui >}}Active{{< /ui >}} or {{< ui >}}Quarantined{{< /ui >}}.</p>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Limit the exclusion claim to the 30-day automation

Users can incorrectly think that the remediation flow cannot move a disabled test to Fixed.

Assertion details
  • Input: A disabled test completes the remediation flow, all retries pass, and the fix branch merges.
  • Expected: The text must limit the exclusion to the 30-day no-flakes automation.
  • Actual: The text says that disabled tests never move to Fixed automatically. The remediation flow does this after all retries pass and the fix branch merges.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session

<p>Before Datadog automatically moves a flaky test to {{< ui >}}Fixed{{< /ui >}}, it checks whether the test may be broken rather than fixed. A broken test is a flaky test whose recent executions all failed, resulting in a 100% failure rate over the last 7 days. Datadog does not automatically mark these tests as fixed, which helps prevent quarantined tests that still fail from breaking CI again.</p>
<p>Use the {{< ui >}}Broken test{{< /ui >}} facet in the Flaky Tests Management explorer to identify these tests. Filter on <code>broken_test:true</code> to show tests with a 100% failure rate over the last 7 days.</p>
</td>
Expand Down Expand Up @@ -243,6 +244,15 @@ To use Flaky Tests Management features, you must use Datadog's native instrument

## Troubleshooting

### Disabled flaky tests are not automatically moved to Fixed

Datadog does not automatically move {{< ui >}}Disabled{{< /ui >}} tests to the {{< ui >}}Fixed{{< /ui >}} state, even after they stop flaking for 30 days. A disabled test is skipped in CI, so Datadog receives no test run data for it and cannot verify whether it is still flaky.

To fix a disabled test, do one of the following:

- Trigger the [attempt-to-fix remediation flow](#confirm-fixes-for-flaky-tests). It retries the test even while it is disabled, and moves it to {{< ui >}}Fixed{{< /ui >}} once the fix is confirmed and merged.
- Manually change the test's state to {{< ui >}}Active{{< /ui >}} or {{< ui >}}Quarantined{{< /ui >}} using the [state drop-down](#change-a-flaky-tests-state). Datadog then automatically moves the test to {{< ui >}}Fixed{{< /ui >}} the next time the automation runs, unless the test flakes again.

### Slack notifications are not delivered

If Slack notifications are not being delivered, check that your notification rule uses the `@slack-ACCOUNT-CHANNEL` format.
Expand Down
Loading