-
Notifications
You must be signed in to change notification settings - Fork 373
Description
Describe the bug
@charris-msft had performed an analysis based on the readme content and noticed that prompts for some servers dont actually call into the tools themselves. Here's a copy of the report -
Executive Summary
Tested 19 service areas covering 40+ individual prompt tests from the Azure MCP Server README. Of the service areas tested:
Status Count Percentage
✅ Fully Passed 13 68%
⚠️ Partial Pass 4 21%
❌ Failed 2 11%
Key Findings: - The subscription parameter is effectively required across nearly all tools but is marked optional in tool schemas — this is the #1 systemic issue - App Service and Container Apps have no first-class listing/read commands, despite being major Azure services - The Foundry model catalog works frictionlessly (no params needed), setting a good UX benchmark - Tools that work are generally well-structured with clean JSON responses
Detailed Results by Service Area
✅ Fully Passed (13 services)
Service Area Prompts Tested Key Observations
1 Subscriptions & Resource Groups 2 Both tools work. group_list requires explicit subscription.
2 Azure Advisor 1 Returned 50 recommendations. Clean JSON response.
3 Azure App Configuration 2 Listed stores and key-value pairs. Two-step workflow needed for KV pairs.
4 Azure CLI Generate 2 Generated correct az group list and az storage account list commands.
5 Azure CLI Install 3 All 3 CLIs (az, azd, func) returned OS-specific install instructions.
6 Azure Compute 2 Listed VMs and VMSS (empty but valid). Schema inconsistency on subscription.
7 Azure Container Registry 1 Listed 46 registries across regions.
8 Azure Cosmos DB 2 Listed accounts, databases, and containers across subscriptions.
9 Azure Event Grid 1 Listed topics (empty but valid).
10 Azure Kubernetes Service 1 Listed clusters (empty but valid). Node pool test skipped — no clusters.
11 Azure Monitor 5 Full workflow tested: list workspaces → list tables → KQL query. Excellent.
12 Azure Storage 2 Listed 3 accounts, got details for one. Unified list/get pattern works well.
13 Microsoft Foundry 7 Model catalog (no params!), deployments, agents, resources all work.
⚠️ Partial Pass (4 services)
Service Area Issue
14 Azure Key Vault 3/4 prompts passed. keyvault_admin_settings_get only works for Managed HSM, not standard vaults. No keyvault_list command to discover vaults.
15 Azure SQL sql_server_list requires resource-group — cannot list servers subscription-wide. Must iterate through all RGs manually.
16 Azure AI Search Tool chain works but 3/4 search services returned 403 Forbidden (RBAC). search_index_get doubles as list (no separate list command).
17 Azure App Lens Generic prompt works; specific CPU/memory query returned excellent diagnostics. But broad queries return empty arrays without context, and one query hit a transient 401.
❌ Failed (2 services)
Service Area Issue
18 Azure App Service Tool has only 1 command (appservice_database_add). Cannot list web apps, get app details, or list app service plans. Zero read operations.
19 Azure Container Apps No dedicated MCP tool exists. Had to fall back to az containerapp list via CLI generate + shell. 11 container apps found but only via workaround.
Expected behavior
The suggestion is review the existing prompts in ReadMe and align with the prompts that the server/tools support (align with subset of prompts from e2etestprompts).
Alternatively add new commands/tools to support the gaps if needed.
Actual behavior
For example -
The readme suggests that list operation is provided by the mcp tools. This needs to be fixed to what the App Service tools currently supports today. For example here's what is in readme and there are no tools in the App Service server for this today.
App service supports add tools instead -
The suggestion is review the existing prompts in ReadMe and align with the prompts that the server/tools support (align with subset of prompts from e2etestprompts).
Reproduction Steps
Test these prompts via copilot CLI
Environment
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status

