[SDTEST-3826] Extend plan reporting

[anmarchenko]

What

Extend the DDTest plan report with richer planning diagnostics:

• Print DDTest configuration, including min/max parallelism.
• Add a compact warning section for slow suites placed on dedicated runners.
• Print the 10 slowest suites overall with historical duration and estimated runtime.
• Split report data building from report rendering.
• Remove structured logs that duplicated information now printed in the report.

Why

The plan report is the best place for users to understand why a run was split the way it was. Keeping these details in the report makes the output easier to scan and avoids duplicating the same information in logs. The slowest-suite reporting uses bounded top-N collection so it stays small even for large repositories.

E2E testing

Run ddtest plan on a repository with report output enabled and verify the report includes DDTest settings, Datadog settings, planning/split details, slow suites on dedicated runners when applicable, and the 10 slowest test suites overall.

Appendix

Example golden report from internal/planner/report_test.go:

+++ DDTest: plan report

Run
  Service: checkout-api
  Repository: https://github.com/acme/checkout.git
  Commit: 9f3a1c7d2b4e
  Branch: feature/split-report
  Platform: ruby / rspec
  OS tags: os.platform=linux, os.architecture=amd64, os.version=6.8.0
  Runtime tags: runtime.name=ruby, runtime.version=3.3.4

DDTest settings
  Platform: ruby
  Framework: rspec
  Min parallelism: 2
  Max parallelism: 8
  CI job overhead: 25s
  Worker env: DATABASE_PASSWORD, RAILS_ENV
  CI node: -1
  CI node workers: 2
  Command: bundle exec rspec
  Tests location: spec/**/*_spec.rb
  Runtime tags: {"runtime.version":"3.3.4"}
  Report enabled: true

Datadog settings
  Test Impact Analysis: enabled
    Test skipping: enabled
    Test impact collection: disabled
  Known tests: enabled
  Impacted tests: disabled
  Early flake detection: enabled
  Auto test retries: enabled
  Flaky test management: enabled

Backend data
  Known tests: 4 modules, 1,284 suites, 18,921 tests
  Skippable tests for this run: 312
  Managed flaky tests: 26 total, 8 quarantined, 3 disabled, 5 attempt-to-fix

Planning
  Test files discovered: 642
  Fully skipped files: 118
  Test files to run: 524
  Duration source: 431 known, 90 default
  Estimated time saved: 38.40%

Split
  Runners: 6
  Expected wall time: 4m12s
  Imbalance: 11s
  Total estimated runtime: 23m46s

Slow suites on dedicated runners
  ATTENTION: 1 dedicated runner
  1. runner 0, rspec / Checkout::Slow (spec/slow_spec.rb): historical duration 2m0s, estimated runtime 1m40s

10 slowest test suites overall
  1. rspec / Checkout::VerySlow (spec/very_slow_spec.rb): historical duration 3m0s, estimated runtime 3m0s
  2. rspec / Checkout::Slow (spec/slow_spec.rb): historical duration 2m0s, estimated runtime 1m40s

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f149496587

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[anmarchenko]

E2E Test Report: SUCCESS

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local Shepherd/crook plan-only testing with Forem (ruby / rspec)
• PR: [SDTEST-3826] Extend plan reporting #76
• Revision tested: 064488853725f75519a27e761023bd6641ed1348
• Branch tested: anmarchenko/improve-plan-reporting
• Playground: forem
• Scenario: ddtest-suite-durations-forem-policies
• Command preset: ddtest-plan
• Target: mockdog

Results

Check	Status	Evidence
PR branch built and injected by crook	PASS	ddtest @ anmarchenko/improve-plan-reporting
Report is enabled and printed	PASS	+++ DDTest: plan report, Report enabled: true
DDTest settings section is present	PASS	Platform, framework, min/max parallelism, overhead, worker env, tests location
Datadog settings section is present	PASS	TIA, known tests, EFD, ATR, test management statuses
Backend/planning/split details are present	PASS	23 files discovered, 23 known durations, 3 runners
Slow dedicated-runner warning is present	PASS	ATTENTION: 1 dedicated runner
10 slowest suites are printed in expected order	PASS	ArticlePolicy 10s, CommentPolicy 8s, ResponseTemplatePolicy 5s, then 100ms suites
Worker env values are redacted	PASS	Report printed keys only: DATABASE_PASSWORD, RAILS_ENV, TOKEN

Issues Found

None.

Observed expected Forem planning warnings because TIA/test skipping were disabled by the mockdog scenario, so full discovery was cancelled and ddtest used fast file discovery. The fallback discovered all 23 policy spec files, and mocked suite durations were fetched successfully.

Verification

• Mockdog/backend verification: PASS. POST /api/v2/ci/ddtest/test_suite_durations returned 23 suites from the scenario.
• Plan output verification: PASS. The generated .testoptimization stash had parallel-runners.txt = 3, test-files.txt = 23, and runner split sizes 1 / 1 / 21.
• Datadog UI verification: Not applicable for this PR/run. This change affects ddtest plan report rendering; the validation was plan-only against mockdog and did not execute tests or emit Test Visibility sessions.

Test Methodology

1. Fetched PR metadata and confirmed the E2E request was to run ddtest plan with report output enabled.
2. Used local docs for test parallelization and selected Forem with ddtest-suite-durations-forem-policies, which has three heavy policy specs and enough timing data to exercise slow-suite reporting.
3. Ran crook with --dep ddtest=anmarchenko/improve-plan-reporting, --debug, and DD_TEST_OPTIMIZATION_RUNNER_TESTS_LOCATION=spec/policies/*_spec.rb.
4. Ran with DD_TEST_OPTIMIZATION_RUNNER_CI_JOB_OVERHEAD=0s to make the selector choose multiple runners locally and exercise the dedicated-runner report path.
5. Ran an additional plan-only pass with worker env values to verify that the report prints only worker env keys and does not expose values.
6. Inspected logs and stashed .testoptimization planning files.

Full ddtest Report Output

+++ DDTest: plan report

Run
  Service: forem
  Repository: git@github.com:anmarchenko/forem.git
  Commit: 3ff8dd4a1a3bc09f39b7c94fcfdcf819d82477c2
  Branch: main
  Platform: ruby / rspec
  OS tags: os.platform=darwin23, os.architecture=arm64, os.version=25.5.0
  Runtime tags: runtime.name=ruby, runtime.version=3.3.0

DDTest settings
  Platform: ruby
  Framework: rspec
  Min parallelism: 1
  Max parallelism: 8
  CI job overhead: 0s
  Worker env: DATABASE_PASSWORD, RAILS_ENV, TOKEN
  CI node: -1
  CI node workers: 1
  Command: not set
  Tests location: spec/policies/*_spec.rb
  Runtime tags: not set
  Report enabled: true

Datadog settings
  Test Impact Analysis: disabled
    Test skipping: disabled
    Test impact collection: disabled
  Known tests: disabled
  Impacted tests: disabled
  Early flake detection: disabled
  Auto test retries: disabled
  Flaky test management: disabled

Backend data
  Known tests: disabled
  Skippable tests for this run: disabled
  Managed flaky tests: disabled

Planning
  Test files discovered: 23
  Fully skipped files: 0
  Test files to run: 23
  Duration source: 23 known, 0 default
  Estimated time saved: 0.00%

Split
  Runners: 3
  Expected wall time: 10s
  Imbalance: 3s
  Total estimated runtime: 25s

Slow suites on dedicated runners
  ATTENTION: 1 dedicated runner
  1. runner 0, rspec / ArticlePolicy (spec/policies/article_policy_spec.rb): historical duration 10s, estimated runtime 10s

10 slowest test suites overall
  1. rspec / ArticlePolicy (spec/policies/article_policy_spec.rb): historical duration 10s, estimated runtime 10s
  2. rspec / CommentPolicy (spec/policies/comment_policy_spec.rb): historical duration 8s, estimated runtime 8s
  3. rspec / ResponseTemplatePolicy (spec/policies/response_template_policy_spec.rb): historical duration 5s, estimated runtime 5s
  4. rspec / AdminPolicy (spec/policies/admin_policy_spec.rb): historical duration 100ms, estimated runtime 100ms
  5. rspec / ApiSecretPolicy (spec/policies/api_secret_policy_spec.rb): historical duration 100ms, estimated runtime 100ms
  6. rspec / ApplicationPolicy (spec/policies/application_policy_spec.rb): historical duration 100ms, estimated runtime 100ms
  7. rspec / Authorizer (spec/policies/authorizer_spec.rb): historical duration 100ms, estimated runtime 100ms
  8. rspec / DiscussionLockPolicy (spec/policies/discussion_lock_policy_spec.rb): historical duration 100ms, estimated runtime 100ms
  9. rspec / FollowPolicy (spec/policies/follow_policy_spec.rb): historical duration 100ms, estimated runtime 100ms
  10. rspec / GithubRepoPolicy (spec/policies/github_repo_policy_spec.rb): historical duration 100ms, estimated runtime 100ms

---

This E2E test was performed by Shepherd - autonomous QA agent for Datadog Test Optimization.