Add composable deterministic task validation

[luisorofino]

This is PR 1 of 3 in a stacked series:

1. Add composable deterministic task validation #24777 (reusable validation framework) <- you are here
2. Add OpenMetrics label hygiene validation #24778 (OpenMetrics label-hygiene validation)
3. Add TUI feedback for validations #24788 (Adds TUI feedback for deterministic checks)

What does this PR do?

Adds composable deterministic validation to agentic tasks without requiring an LLM reviewer.

• Lets phase subclasses declare any number of named deterministic checks through deterministic_checks().
• Runs all deterministic checks in declaration order and combines their failures into one actionable repair prompt.
• Supports deterministic-only validation, reviewer-only validation, and deterministic checks followed by an optional goal reviewer.
• Creates the reviewer lazily, only after deterministic checks pass and only when the task has a goal.
• Generalizes validation attempts and checkpoint records from goal-specific names to task-validation names.
• Renames max_goal_attempts to max_validation_attempts in the task model and existing flow configuration.

The OpenMetrics-specific check and the TUI updates are intentionally kept out of this PR (see the stack).

Motivation

AI-7081 describes a seven-minute review-and-retry round trip caused by a deterministic label-policy miss. The framework previously coupled validation to an LLM goal reviewer, which meant phases could not cheaply reject artifacts using local, repeatable checks.

This layer introduces the generic composition and retry lifecycle so future phases can add fast checks without forcing a reviewer agent or duplicating orchestration logic.

Scoped coverage includes validation composition, deterministic-only repair and exhaustion, lazy reviewer construction, combined deterministic-plus-reviewer validation, checkpoint recording, configuration validation, and existing orchestration behavior. The focused suite passed 230 tests; Ruff and mypy also pass.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-datadog-prod-us1-2]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 98.71%
• Overall Coverage: 89.09%

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: dd778b6 | Docs | Datadog PR Page | Give us feedback!}

[luisorofino]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b02384afe6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run