diff --git a/hugo/content/en/tests/flaky_management/_index.md b/hugo/content/en/tests/flaky_management/_index.md index 82a8d831986..fbff81fd947 100644 --- a/hugo/content/en/tests/flaky_management/_index.md +++ b/hugo/content/en/tests/flaky_management/_index.md @@ -34,7 +34,7 @@ Use the state drop-down to change how a flaky test is handled in your CI pipelin | ----------- | ----------- | | {{< ui >}}Active{{< /ui >}} | The test is known to be flaky and is running in CI. | | {{< ui >}}Quarantined{{< /ui >}} | Keep the test running in the background, but failures don't affect CI status or break pipelines. This is useful for isolating flaky tests without blocking merges. Datadog tags test run events with `@test.test_management.is_quarantined:true` when quarantined. | -| {{< ui >}}Disabled{{< /ui >}} | Skip the test entirely in CI. Use this when a test is no longer relevant or needs to be temporarily removed from the pipeline. Datadog tags test run events with `@test.test_management.is_disabled:true` when disabled. | +| {{< ui >}}Disabled{{< /ui >}} | Skip the test entirely in CI. Use this when a test is no longer relevant or needs to be temporarily removed from the pipeline. Datadog tags test run events with `@test.test_management.is_disabled:true` when disabled. Disabled tests are excluded from the automatic move to {{< ui >}}Fixed{{< /ui >}}. | | {{< ui >}}Fixed{{< /ui >}} | The test has passed consistently and is no longer flaky. If supported, use the [remediation flow](#confirm-fixes-for-flaky-tests) to confirm the fix and automatically apply this state after it is merged into the default branch. |
State actions have minimum version requirements for each programming language's instrumentation library. See Compatibility for details.
@@ -88,6 +88,7 @@ Configure automated Flaky Test Policies to govern how flaky tests are handled in {{< ui >}}Fixed{{< /ui >}}

If a flaky test no longer flakes for 30 days, it is automatically moved to the Fixed state. This automation is default behavior and can't be customized.

+

Tests in the {{< ui >}}Disabled{{< /ui >}} state are excluded from this automation and are never automatically moved to {{< ui >}}Fixed{{< /ui >}}. Because disabled tests are skipped in CI, Datadog has no signal about whether they are still flaky. To make a disabled test eligible again, change its state to {{< ui >}}Active{{< /ui >}} or {{< ui >}}Quarantined{{< /ui >}}.

Before Datadog automatically moves a flaky test to {{< ui >}}Fixed{{< /ui >}}, it checks whether the test may be broken rather than fixed. A broken test is a flaky test whose recent executions all failed, resulting in a 100% failure rate over the last 7 days. Datadog does not automatically mark these tests as fixed, which helps prevent quarantined tests that still fail from breaking CI again.

Use the {{< ui >}}Broken test{{< /ui >}} facet in the Flaky Tests Management explorer to identify these tests. Filter on broken_test:true to show tests with a 100% failure rate over the last 7 days.

@@ -243,6 +244,15 @@ To use Flaky Tests Management features, you must use Datadog's native instrument ## Troubleshooting +### Disabled flaky tests are not automatically moved to Fixed + +Datadog does not automatically move {{< ui >}}Disabled{{< /ui >}} tests to the {{< ui >}}Fixed{{< /ui >}} state, even after they stop flaking for 30 days. A disabled test is skipped in CI, so Datadog receives no test run data for it and cannot verify whether it is still flaky. + +To fix a disabled test, do one of the following: + +- Trigger the [attempt-to-fix remediation flow](#confirm-fixes-for-flaky-tests). It retries the test even while it is disabled, and moves it to {{< ui >}}Fixed{{< /ui >}} once the fix is confirmed and merged. +- Manually change the test's state to {{< ui >}}Active{{< /ui >}} or {{< ui >}}Quarantined{{< /ui >}} using the [state drop-down](#change-a-flaky-tests-state). Datadog then automatically moves the test to {{< ui >}}Fixed{{< /ui >}} the next time the automation runs, unless the test flakes again. + ### Slack notifications are not delivered If Slack notifications are not being delivered, check that your notification rule uses the `@slack-ACCOUNT-CHANNEL` format.