[EPIC] Experiment-Driven Tool Evaluation

[LarryOsterman]

Problem statement

Azure MCP exposes baseline, namespace, and consolidated tool surfaces whose quality cannot be judged from implementation alone. Tool descriptions, routing, prompt coverage, confidence, and end-to-end task success need reproducible experiments so product decisions are based on measured agent behavior rather than isolated examples.

Vision

Every supported Azure MCP tool mode is evaluated through repeatable experiments with realistic tasks, structured results, dashboards, and CI signals that identify regressions and guide product decisions.

Current work

• Tool Evaluations: Flesh out existing evaluation experiments to include more tool coverage #3036, expand experiments across more tools and service areas
• Tool Evaluations: Update dashboards to report evaluation experiment results #3037, report experiment results through dashboards
  • Tool Evaluations: Add CI pipeline support for evaluation experiments #3038, add CI pipeline support for evaluation experiments
• Tool Evaluations: Create task-based holistic prompts for variant comparison #3039, create task-based holistic prompts for variant comparison
• Integrate vally into CI #2848, integrate Vally into CI

Related enforcement work such as #2497 is consumed through the PR quality guardrails epic.

Completed foundation

• MQ: Evaluate using Microsoft.Extensions.AI.Evaluation tool for e2etests automation #238 evaluated Microsoft.Extensions.AI.Evaluation
• Centralize Utility methods in ToolDescriptionEvaluator and ToolMetadataExporter #1506 centralized ToolDescriptionEvaluator utilities
• Add Managed Identity support in Tool Description Evaluator #471 added managed-identity support to the evaluator
• The existing runner and prompt sets provide the baseline for namespace and consolidated comparisons

Goals (in scope)

• Compare baseline, namespace, and consolidated modes using consistent datasets and metrics
• Expand prompt and tool coverage across representative Azure scenarios
• Add realistic multi-step and task-level evaluations beyond isolated tool matching
• Emit structured experiment results suitable for CI and dashboards
• Define regression thresholds and ownership for failed evaluations
• Use results to guide tool descriptions, routing, and mode decisions

Non-goals (out of scope)

• Treating embedding similarity as the only product-quality signal
• Owning implementation of routing or protocol behavior being measured
• Blocking every pull request on experimental or statistically insignificant variation

Success criteria

• Variant experiments run reproducibly against a documented baseline
• Representative service areas and holistic tasks have maintained datasets
• CI and dashboards expose comparable quality and confidence trends
• Material regressions produce actionable failures with responsible owners
• Product mode and routing decisions cite measured experiment results

Dependencies

• Stable server modes and generated tool definitions
• Evaluation models, credentials, datasets, and Vally integration
• CI capacity and durable result storage
• PR guardrails that consume established evaluation thresholds