Skip to content

[BUG] Readme prompts dont align with supported tools #1726

@kvenkatrajan

Description

@kvenkatrajan

Describe the bug

@charris-msft had performed an analysis based on the readme content and noticed that prompts for some servers dont actually call into the tools themselves. Here's a copy of the report -

Executive Summary
Tested 19 service areas covering 40+ individual prompt tests from the Azure MCP Server README. Of the service areas tested:
Status Count Percentage
✅ Fully Passed 13 68%
⚠️ Partial Pass 4 21%
❌ Failed 2 11%
Key Findings: - The subscription parameter is effectively required across nearly all tools but is marked optional in tool schemas — this is the #1 systemic issue - App Service and Container Apps have no first-class listing/read commands, despite being major Azure services - The Foundry model catalog works frictionlessly (no params needed), setting a good UX benchmark - Tools that work are generally well-structured with clean JSON responses
Detailed Results by Service Area
✅ Fully Passed (13 services)
Service Area Prompts Tested Key Observations
1 Subscriptions & Resource Groups 2 Both tools work. group_list requires explicit subscription.
2 Azure Advisor 1 Returned 50 recommendations. Clean JSON response.
3 Azure App Configuration 2 Listed stores and key-value pairs. Two-step workflow needed for KV pairs.
4 Azure CLI Generate 2 Generated correct az group list and az storage account list commands.
5 Azure CLI Install 3 All 3 CLIs (az, azd, func) returned OS-specific install instructions.
6 Azure Compute 2 Listed VMs and VMSS (empty but valid). Schema inconsistency on subscription.
7 Azure Container Registry 1 Listed 46 registries across regions.
8 Azure Cosmos DB 2 Listed accounts, databases, and containers across subscriptions.
9 Azure Event Grid 1 Listed topics (empty but valid).
10 Azure Kubernetes Service 1 Listed clusters (empty but valid). Node pool test skipped — no clusters.
11 Azure Monitor 5 Full workflow tested: list workspaces → list tables → KQL query. Excellent.
12 Azure Storage 2 Listed 3 accounts, got details for one. Unified list/get pattern works well.
13 Microsoft Foundry 7 Model catalog (no params!), deployments, agents, resources all work.
⚠️ Partial Pass (4 services)
Service Area Issue
14 Azure Key Vault 3/4 prompts passed. keyvault_admin_settings_get only works for Managed HSM, not standard vaults. No keyvault_list command to discover vaults.
15 Azure SQL sql_server_list requires resource-group — cannot list servers subscription-wide. Must iterate through all RGs manually.
16 Azure AI Search Tool chain works but 3/4 search services returned 403 Forbidden (RBAC). search_index_get doubles as list (no separate list command).
17 Azure App Lens Generic prompt works; specific CPU/memory query returned excellent diagnostics. But broad queries return empty arrays without context, and one query hit a transient 401.
❌ Failed (2 services)
Service Area Issue
18 Azure App Service Tool has only 1 command (appservice_database_add). Cannot list web apps, get app details, or list app service plans. Zero read operations.
19 Azure Container Apps No dedicated MCP tool exists. Had to fall back to az containerapp list via CLI generate + shell. 11 container apps found but only via workaround.

Expected behavior

The suggestion is review the existing prompts in ReadMe and align with the prompts that the server/tools support (align with subset of prompts from e2etestprompts).

Alternatively add new commands/tools to support the gaps if needed.

Actual behavior

For example -
The readme suggests that list operation is provided by the mcp tools. This needs to be fixed to what the App Service tools currently supports today. For example here's what is in readme and there are no tools in the App Service server for this today.

Image

App service supports add tools instead -

Image

The suggestion is review the existing prompts in ReadMe and align with the prompts that the server/tools support (align with subset of prompts from e2etestprompts).

Reproduction Steps

Test these prompts via copilot CLI

Environment

No response

Metadata

Metadata

Labels

needs-team-attentionWorkflow: This issue needs attention from Azure service team or MCP team.server-Azure.McpAzure.Mcp.Server

Type

No type

Projects

Status

Not Started

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions