[NO TICKET] Document that disabled flaky tests are excluded from auto-fix - #39657
Merged
TovRudyy merged 2 commits intoSep 2, 2026
Merged
Conversation
Clarify in the flaky test management docs that tests in the Disabled state are never automatically moved to Fixed, explain why, and how to re-enable. This matches the in-app warnings that link here. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
TovRudyy
marked this pull request as ready for review
September 2, 2026 12:29
Contributor
Preview links (active after the
|
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
The new text says that a disabled test never moves to Fixed automatically. The existing remediation flow does this after a successful fix merges.
🤖 Datadog Autotest · Commit 907efc8 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| <td>{{< ui >}}Fixed{{< /ui >}}</td> | ||
| <td> | ||
| <p>If a flaky test no longer flakes for 30 days, it is automatically moved to the Fixed state. This automation is default behavior and can't be customized.</p> | ||
| <p>Tests in the {{< ui >}}Disabled{{< /ui >}} state are excluded from this automation and are never automatically moved to {{< ui >}}Fixed{{< /ui >}}. Because disabled tests are skipped in CI, Datadog has no signal about whether they are still flaky. To make a disabled test eligible again, change its state to {{< ui >}}Active{{< /ui >}} or {{< ui >}}Quarantined{{< /ui >}}.</p> |
Contributor
There was a problem hiding this comment.
Users can incorrectly think that the remediation flow cannot move a disabled test to Fixed.
Assertion details
- Input: A disabled test completes the remediation flow, all retries pass, and the fix branch merges.
- Expected:
The text must limit the exclusion to the 30-day no-flakes automation. - Actual:
The text says that disabled tests never move to Fixed automatically. The remediation flow does this after all retries pass and the fix branch merges.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest · Open Bits AI session
juan-fernandez
approved these changes
Sep 2, 2026
Explain why disabled flaky tests are never automatically moved to Fixed and give the two ways to fix them: trigger the attempt-to-fix remediation flow, or manually change the state back to Active or Quarantined. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
estherk15
approved these changes
Sep 2, 2026
estherk15
left a comment
estherk15
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the update!
TovRudyy
deleted the
olek.rudyy/document-disabled-flaky-tests-excluded-from-auto-fix
branch
September 2, 2026 13:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
Clarify in the Flaky Test Management docs that tests in the Disabled state are excluded from the automatic move to Fixed. Disabled tests are always skipped in CI, so Datadog has no signal about whether they are still flaky, and therefore never auto-moves them to Fixed.
The clarification is added in two places: the Fixed automation row (where the auto-move behavior is described and where the in-app link lands) with the full explanation and how to re-enable, and a short note on the Disabled state definition in the state table.
Merge readiness
For Datadog employees:
<name>/<description>convention and include the forward slash (/).AI assistance
Used Claude Code to locate the relevant docs section (matching the anchor the Datadog UI links to) and draft the wording.
Additional notes
The
#configure-policies-to-automate-the-flaky-test-lifecycleheading text is unchanged, so existing in-app deep links keep resolving.