Skip to content

feat(review): surface the deterministic test-evidence classifier in the AI review prompt #2558

Description

@JSONbored

Parent: #1936

Problem

src/signals/test-evidence.ts already computes, per changed path, whether test evidence exists (hasLocalTestEvidence/isTestPath), and src/signals/slop.ts uses it to build a missingTestEvidence slop finding. But src/services/ai-review.ts's buildUserPrompt and REVIEW_SYSTEM_PROMPT never reference this classification at all. The AI reviewer therefore judges test adequacy purely by eye on the raw diff, with no access to the engine's own deterministic classification of which changed files/branches lack corresponding test files. This is distinct from Codecov's post-CI line/branch percentage — this classifier could tell the model, before it writes its verdict, exactly which specific changed files have zero test-path evidence, so the reviewer's blockers/nits can name them explicitly instead of guessing generically.

Requirements

  • Splice a "changed files with zero test-path evidence" section into the AI review user prompt, reusing the existing test-evidence.ts classifier — no new signal/classifier needed, this is a wiring gap only.
  • Keep the section concise (file list, not full content) to avoid meaningfully growing the prompt token budget.
  • Must not change existing blocker/nit taxonomy — this is additional CONTEXT for the model, not a new deterministic rule.

Deliverables

  • buildUserPrompt in src/services/ai-review.ts includes a test-evidence section sourced from src/signals/test-evidence.ts.
  • A test confirming the section appears in the built prompt when applicable changed files lack test evidence, and is absent/empty when all changed files have coverage.

Acceptance criteria

  • The AI review prompt for a PR touching files with zero test-path evidence includes that information explicitly.
  • No regression to existing prompt construction/token budget behavior for PRs where test evidence is present.

Expected outcome

The AI reviewer's test-adequacy judgment is grounded in the engine's own deterministic classification instead of purely eyeballing the diff — a small, cheap change that should measurably reduce missed "no tests for this change" findings.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions