PDX-470/471/473: feat(mcp): add detail level, diff mode, and completeness score to validation tools

[mrdailey99]

Summary

• PDX-470: Add detail=summary|standard|full parameter to all four structured-response validation tools (provar_testsuite_validate, provar_testplan_validate, provar_testcase_validate, provar_project_validate). summary returns only key scores and the stop signal — eliminates token explosion on repeat calls.
• PDX-471: Add baseline_run_id diff mode to provar_testsuite_validate and provar_project_validate. Each validation call saves a violation snapshot keyed by run_id; subsequent calls with baseline_run_id return only {added, resolved, unchanged_count} instead of the full inventory.
• PDX-473: Add completeness_score (0–100) and recommended_next_action (stop | inspect_failures | fix_and_revalidate) to all four tools. Agents can use recommended_next_action=stop as an unambiguous signal to end the fix-validate loop.

New shared utilities

File	Exports
src/mcp/utils/detailLevel.ts	applyDetailLevel, DetailLevel
src/mcp/utils/validationScore.ts	calcCompletenessScore, calcNextAction
src/mcp/utils/validationDiff.ts	generateRunId, saveRun, hasAnyRun, loadBaselineViolations, computeDiff

Test plan

• validationScore.test.ts — 8 tests for calcCompletenessScore and calcNextAction
• validationDiff.test.ts — 13 tests for run persistence, cap-at-20 eviction, and diff computation
• testSuiteValidate.test.ts — 30 tests (19 existing + 11 new for PDX-470/471/473)
• testPlanValidate.test.ts — 28 tests (19 existing + 9 new for PDX-470/473)
• All 79 tests pass locally in the isolated worktree

🤖 Generated with Claude Code

[github-actions]

Quality Orchestrator

🟢 LOW · 24 / 100 · Touches: utils. All changed files have mapped tests.

---

🧪 Tests to Run · Running 7 of 47 tests

• unit/mcp/projectValidateFromPath.test.ts
• unit/mcp/testCaseValidate.test.ts
• unit/mcp/testPlanValidate.test.ts
• unit/mcp/testSuiteValidate.test.ts
• unit/mcp/detailLevel.test.ts
• unit/mcp/validationDiff.test.ts
• unit/mcp/validationScore.test.ts

▶ Run command

npx vitest run \
  unit/mcp/projectValidateFromPath.test.ts \
  unit/mcp/testCaseValidate.test.ts \
  unit/mcp/testPlanValidate.test.ts \
  unit/mcp/testSuiteValidate.test.ts \
  unit/mcp/detailLevel.test.ts \
  unit/mcp/validationDiff.test.ts \
  unit/mcp/validationScore.test.ts

---

_{⚡ quality-orchestrator  ·  /qo stub <file>  ·  qo analyze-local}

[Copilot]

Pull request overview

Adds response shaping, validation scoring, and diff-run persistence utilities for MCP validation tools so agents can request smaller responses, compare runs, and decide whether to continue validation loops.

Changes:

• Adds detail response levels and completeness_score / recommended_next_action fields across validation tools.
• Adds run snapshot persistence and diff computation utilities, with baseline diff mode wired into suite, testcase, and project validation.
• Adds unit tests for score/diff utilities plus suite/plan response changes.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
src/mcp/utils/detailLevel.ts	Adds shared summary/standard/full response filtering helper.
src/mcp/utils/validationScore.ts	Adds completeness score and next-action helpers.
src/mcp/utils/validationDiff.ts	Adds run id generation, snapshot persistence, baseline loading, and diff computation.
src/mcp/tools/testSuiteValidate.ts	Wires detail level, run ids, diff mode, and next-action scoring into suite validation.
src/mcp/tools/testPlanValidate.ts	Wires detail level and next-action scoring into plan validation.
src/mcp/tools/testCaseValidate.ts	Wires detail level, run ids, diff mode, and next-action scoring into testcase validation.
src/mcp/tools/projectValidateFromPath.ts	Wires detail level, run ids, diff mode, and next-action scoring into project validation.
src/mcp/tools/descHelper.ts	Adds an environment-driven description helper.
test/unit/mcp/validationScore.test.ts	Adds unit coverage for scoring and next-action helpers.
test/unit/mcp/validationDiff.test.ts	Adds unit coverage for run persistence and diff helper behavior.
test/unit/mcp/testSuiteValidate.test.ts	Adds suite validation tests for detail, score/action, and baseline diff behavior.
test/unit/mcp/testPlanValidate.test.ts	Adds plan validation tests for detail and score/action behavior.

Comments suppressed due to low confidence (3)

src/mcp/tools/testSuiteValidate.ts:172

• hasAnyRun(storageDir) is evaluated after saveRun() has already written the current run, so it will be true even on the first validation for this suite. A failing first run will therefore return fix_and_revalidate instead of the intended inspect_failures first-run action.

        const completeness_score = calcCompletenessScore(summary.test_cases_valid, summary.total_test_cases);
        const hasBaseline = hasAnyRun(storageDir);
        const recommended_next_action = calcNextAction(completeness_score, hasBaseline);

src/mcp/tools/testCaseValidate.ts:190

• The current run is persisted before loading baseline_run_id. When the requested baseline is the oldest retained run, this write can trigger eviction at the 20-run cap and make an otherwise valid baseline return BASELINE_NOT_FOUND.

        try {
          saveRun(storageDir, runId, currentViolations);
        } catch (saveErr) {
          log('warn', 'provar_testcase_validate: could not save run for diff', {
            requestId,
            error: (saveErr as Error).message,
          });
        }

        // Diff mode
        if (baseline_run_id !== undefined && baseline_run_id !== '') {
          const baseline = loadBaselineViolations(storageDir, baseline_run_id);

src/mcp/tools/projectValidateFromPath.ts:264

• The current project run is saved before baseline_run_id is loaded. If the supplied baseline is the oldest of the 20 retained runs, saving the new run can evict it first, so a valid baseline id fails with BASELINE_NOT_FOUND.

        if (save_results !== false) {
          try {
            saveRun(storageDir, runId, currentViolations);
          } catch (saveErr) {
            log('warn', 'provar_project_validate: could not save run for diff', {
              requestId,
              error: (saveErr as Error).message,
            });
          }
        }

        // Diff mode
        if (baseline_run_id !== undefined && baseline_run_id !== '') {
          const baseline = loadBaselineViolations(storageDir, baseline_run_id);

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.