Add Vally evaluation harness for Event Hubs tools

[LarryOsterman]

Summary

Adds a first Vally evaluation harness under servers/Azure.Mcp.Server/tests/Vally, covering the eventhubs_eventhub_get and eventhubs_namespace_get tools.

• Invoke-VallyEval.ps1: discovery-based runner that walks the directory for evaluations organized by area (namespace subfolder) and tool (.experiment.yaml).
• Each experiment runs a shared eval spec (.eval.yaml) as two variants -- baseline (without the Azure MCP server) and treatment (with it) -- to isolate the server's contribution, graded identically via outcome-based LLM-judge graders.
• eventhubs/New-EventHubsResources.ps1 / Remove-EventHubsResources.ps1: provisioning scripts for the Event Hubs area (tagged with DeleteAfter, local auth disabled for Safe Secrets Standard compliance).
• Updated .vscode/cspell.json with new dictionary terms and CHANGELOG.md with an entry under Other Changes.

This content was migrated from the larryo/first_vally_evaluations branch in the fork larryosterman/mcp.

Depends on #3187

Invoking Livetests

Copilot submitted PRs are not trustworthy by default. Users with write access to the repo need to validate the contents of this PR before leaving a comment with the text /azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Adds a Vally-based evaluation harness under servers/Azure.Mcp.Server/tests/Vally to measure agent effectiveness for Azure MCP Event Hubs tools using baseline vs treatment experiment variants, including optional per-area Azure provisioning/teardown scripts.

Changes:

• Added a discovery-based PowerShell runner (Invoke-VallyEval.ps1) to build azmcp, discover experiments, provision resources per area, run experiments (optionally multiple iterations), and summarize baseline vs treatment outcomes.
• Added Event Hubs evaluation specs (*.eval.yaml) and experiments (*.experiment.yaml) for eventhubs_eventhub_get and eventhubs_namespace_get, plus provisioning/teardown scripts using Azure CLI.
• Updated repo metadata (CHANGELOG.md, .vscode/cspell.json, and a local .gitignore) to document and support the new harness.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
servers/Azure.Mcp.Server/tests/Vally/README.md	Documents the Vally harness layout, runner usage, and how baseline vs treatment comparisons are interpreted.
servers/Azure.Mcp.Server/tests/Vally/Invoke-VallyEval.ps1	Implements experiment discovery, build/PATH setup, optional provisioning, execution, and results summarization.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/New-EventHubsResources.ps1	Provisions Event Hubs resources used by the eval prompts, with DeleteAfter tagging.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/Remove-EventHubsResources.ps1	Tears down provisioned resources by deleting the resource group.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.experiment.yaml	Defines baseline vs treatment variants for the Event Hub “get” capability.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.eval.yaml	Defines stimuli and outcome-based LLM judge grading for Event Hub “get” scenarios.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.experiment.yaml	Defines baseline vs treatment variants for the namespace “get/list” capability.
servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.eval.yaml	Defines stimuli and outcome-based LLM judge grading for namespace “get/list” scenarios.
servers/Azure.Mcp.Server/tests/Vally/.gitignore	Ignores local Vally result artifacts directories.
servers/Azure.Mcp.Server/CHANGELOG.md	Adds an “Other Changes” entry documenting the new Vally harness and initial experiments.
.vscode/cspell.json	Adds new dictionary words related to Vally and Event Hubs eval content.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-consumergroup-get.eval.yaml:132

• In this grader config, scoring: binary is currently indented as part of the prompt block scalar, so it becomes literal prompt text instead of a YAML field. That will leave the grader without an explicit scoring mode (and also pollute the judge prompt). Move scoring: binary out of the prompt block to be a sibling of prompt: under config:.

            Score 1 only if concrete consumer group details obtained from
            Azure are present. Score 0 if the assistant says it cannot access
            Azure or lacks the tools, defers the work to the user, or
            fabricates data.
            scoring: binary

servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.eval.yaml:101

• The grader instruction text has an unintended large whitespace gap (defers the work), which reads like a copy/paste artifact and can also slightly degrade judge prompt clarity. Collapse this to a single space.

            present. Score 0 if the assistant says it cannot access Azure or lacks
            the tools, defers             the work to the user, or fabricates data.
          scoring: binary

servers/Azure.Mcp.Server/tests/Vally/eventhubs/New-EventHubsResources.ps1:99

• This script defines Assert-Az with a comment stating every az call must be followed by an exit-code check, but az account show is not currently checked. If az account show fails (e.g. expired login, bad subscription), $LASTEXITCODE will be non-zero but the script will proceed to JSON parsing/$account checks without surfacing the underlying failure reliably. Capture the JSON, call Assert-Az, then parse.

# Ensure a subscription context exists.
$account = az account show @subArgs 2>$null | ConvertFrom-Json
if (-not $account) {
    throw "No Azure CLI session found. Run 'az login' (and optionally pass -Subscription) first."
}
Write-Info "Using subscription: $($account.name) ($($account.id))"

servers/Azure.Mcp.Server/CHANGELOG.md:161

• This changelog entry says each experiment runs as "two variants—baseline and treatment", but the added evals/runner consistently use three variants (baseline/namespace/consolidated). It also describes the first experiment as only covering eventhubs_eventhub_get and eventhubs_namespace_get, while the PR adds experiments for additional Event Hubs tools (namespace/eventhub/consumer group get/update/delete). Update the changelog wording so it accurately reflects what's in the repo.

- Added a [vally](https://microsoft.github.io/vally) evaluation harness under `servers/Azure.Mcp.Server/tests/Vally` with a first experiment for the `eventhubs_eventhub_get` and `eventhubs_namespace_get` tools. A discovery-based runner script (`Invoke-VallyEval.ps1`) walks the directory for evaluations organized by area (namespace subfolder) and tool (`<tool>.experiment.yaml`). Each experiment runs a shared eval spec (`<tool>.eval.yaml`) as two variants—baseline (without the Azure MCP server) and treatment (with it)—to isolate the server's contribution. Both variants are graded identically on task outcome using outcome-based LLM-judge graders, ensuring a fair comparison regardless of whether the baseline succeeds through other means (e.g. Azure CLI). Each area may supply `New-*Resources.ps1` / `Remove-*Resources.ps1` provisioning scripts (auto-discovered and run per area); the Event Hubs pair creates the required resources (tagged with `DeleteAfter` as a clean-up safety net, with local auth disabled for Safe Secrets Standard compliance) and deletes them afterwards.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 30 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.eval.yaml:102

• The grader instruction text contains an accidental large whitespace run ("defers the work"), which is likely a copy/paste artifact and makes the prompt harder to read.

          prompt: >-
            Score whether the assistant actually retrieved and returned the real
            Event Hubs for the requested namespace and resource group, per the
            rubric. Score 1 only if concrete Event Hubs obtained from Azure are
            present. Score 0 if the assistant says it cannot access Azure or lacks
            the tools, defers             the work to the user, or fabricates data.
          scoring: binary

servers/Azure.Mcp.Server/tests/Vally/README.md:277

• This section says the consolidated report "never" chooses baseline as best, but Invoke-VallyEval.ps1’s Write-ConsolidatedSummary explicitly allows baseline to be selected as best (and prints a "no added value" verdict when it is). Update the README to match the script behavior so readers don’t misinterpret baseline wins as impossible/buggy.

stimulus recorded for that tool into a single set of statistics per variant
(pass rate, and average tokens/turns/wall time/AI credits), names the
best-performing *candidate* variant (highest pass rate first, then the most
efficient on ties - the shared `baseline` control is reported for reference but
never chosen as "best," since it's not a server-mode choice), and lists the

servers/Azure.Mcp.Server/CHANGELOG.md:184

• This changelog entry describes each experiment as running "two variants—baseline and treatment", but the experiments added in this PR define three variants (baseline, namespace, and consolidated). The entry also implies only eventhubs_eventhub_get and eventhubs_namespace_get are covered, but the PR adds additional Event Hubs experiments (update/delete/consumergroup). Please update the changelog text to accurately reflect what was added.

- Added a [vally](https://microsoft.github.io/vally) evaluation harness under `servers/Azure.Mcp.Server/tests/Vally` with a first experiment for the `eventhubs_eventhub_get` and `eventhubs_namespace_get` tools. A discovery-based runner script (`Invoke-VallyEval.ps1`) walks the directory for evaluations organized by area (namespace subfolder) and tool (`<tool>.experiment.yaml`). Each experiment runs a shared eval spec (`<tool>.eval.yaml`) as two variants—baseline (without the Azure MCP server) and treatment (with it)—to isolate the server's contribution. Both variants are graded identically on task outcome using outcome-based LLM-judge graders, ensuring a fair comparison regardless of whether the baseline succeeds through other means (e.g. Azure CLI). Each area may supply `New-*Resources.ps1` / `Remove-*Resources.ps1` provisioning scripts (auto-discovered and run per area); the Event Hubs pair creates the required resources (tagged with `DeleteAfter` as a clean-up safety net, with local auth disabled for Safe Secrets Standard compliance) and deletes them afterwards.