Improve Azure Deploy tool descriptions and e2e test prompts#2023
Merged
g2vinay merged 5 commits intomicrosoft:mainfrom Mar 13, 2026
Merged
Improve Azure Deploy tool descriptions and e2e test prompts#2023g2vinay merged 5 commits intomicrosoft:mainfrom
g2vinay merged 5 commits intomicrosoft:mainfrom
Conversation
Fixes microsoft#1994 - E2E test prompts in Azure Deploy do not trigger corresponding MCP tools in All and Namespace mode. Changes: - DiagramGenerateCommand: Fix concatenation bug (missing spaces between sentences), rewrite description to lead with user-facing purpose and use 'generate/create/visualize architecture diagram' keywords - RulesGetCommand: Replace 'guidelines' language with explicit 'rules and best practices' keywords that match natural user prompts; now scores above bicepschema_get (0.5077 vs 0.4591) - GuidanceGetCommand: Remove 'DO NOT call this tool' negative instruction that suppressed tool selection; lead with CI/CD pipeline creation purpose - GetCommand (Plan): Replace 'Entry point to help the agent' jargon with user-facing 'Creates a deployment plan' language; score improved from 0.5870 (rank microsoft#3) to 0.7173 (rank microsoft#1, tied with actual tool) - e2eTestPrompts.md: Update Azure Deploy test prompts to match the simpler natural-language prompts from the issue report Evaluator improvements (ToolDescriptionEvaluator scores): deploy_architecture_diagram_generate: 0.7260 -> 0.7486 (rank microsoft#2 -> microsoft#1 tie) deploy_iac_rules_get: 0.4312 -> 0.5077 (rank microsoft#2 -> microsoft#1 tie, beats bicepschema_get) deploy_pipeline_guidance_get: 0.6396 -> 0.6614 (rank microsoft#2 -> microsoft#1 tie) deploy_plan_get: 0.5870 -> 0.7173 (rank microsoft#3 -> microsoft#1 tie)
…scores - Append 'Use when user asks: ...' echo phrases to original descriptions for RulesGetCommand, GuidanceGetCommand, and Plan/GetCommand - Update e2eTestPrompts.md with broader prompts that match the echo phrases Scores before/after: deploy_iac_rules_get: 0.508 -> 0.840 deploy_pipeline_guidance_get: 0.661 -> 0.810 deploy_plan_get: 0.714 -> 0.808 Closes microsoft#1994
g2vinay
requested review from
conniey,
fanyang-mono,
hallipr,
jairmyree,
vukelich,
weshaggard and
xiangyan99
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Azure Deploy tool descriptions and the end-to-end test prompt examples to provide clearer “when to use this tool” guidance for agents/users interacting with the Deploy toolset.
Changes:
- Rewrites several Deploy tool
Descriptionstrings to be more explicit about intended usage (plan generation, pipeline guidance, IaC rules, architecture diagram generation). - Updates the Azure Deploy entries in
e2eTestPrompts.mdto match the revised wording/style of the tools.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/Azure.Mcp.Tools.Deploy/src/Commands/Plan/GetCommand.cs | Updates tool description for deployment plan generation. |
| tools/Azure.Mcp.Tools.Deploy/src/Commands/Pipeline/GuidanceGetCommand.cs | Updates tool description for CI/CD pipeline guidance generation. |
| tools/Azure.Mcp.Tools.Deploy/src/Commands/Infrastructure/RulesGetCommand.cs | Updates tool description for IaC rules/best practices retrieval. |
| tools/Azure.Mcp.Tools.Deploy/src/Commands/Architecture/DiagramGenerateCommand.cs | Replaces concatenated description with a multi-line raw string description for diagram generation. |
| servers/Azure.Mcp.Server/docs/e2eTestPrompts.md | Updates Azure Deploy test prompts to align with new tool descriptions. |
joshfree
reviewed
Mar 12, 2026
g2vinay
force-pushed
the
fix/deploy-tool-descriptions
branch
from
c6d997f to
3aaa040
Compare
g2vinay
force-pushed
the
fix/deploy-tool-descriptions
branch
from
75a0b8e to
df56ea6
Compare
vcolin7
approved these changes
Mar 13, 2026
Contributor
vcolin7
left a comment
vcolin7
left a comment
There was a problem hiding this comment.
Just a small comment on the changelog entry. Not a blocker
Co-authored-by: vcolin7 <vicolina@microsoft.com>
wchigit
approved these changes
Mar 13, 2026
github-project-automation
Bot
moved this from Untriaged
to In Progress
in Azure MCP Server
colbytimm
pushed a commit
to colbytimm/microsoft-mcp
that referenced
this pull request
Apr 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improve deploy tool description relevance scores
Fixes #1994. Updates the three Azure Deploy tool descriptions and their corresponding e2eTestPrompts to improve LLM accuracy.
Changes
RulesGetCommand.cs,GuidanceGetCommand.cs, andGetCommand.cs(Plan) — descriptions now include richer, intent-specific vocabulary that better matches real user queriese2eTestPrompts.md— broader, more representative prompts for all three deploy toolsValidation
deploy_iac_rules_getdeploy_pipeline_guidance_getdeploy_plan_get