Skip to content

Update descriptions to ensure deploy area tool invocation#2418

Merged
wchigit merged 14 commits into
mainfrom
wc/update-deploy-invoke
Apr 27, 2026
Merged

Update descriptions to ensure deploy area tool invocation#2418
wchigit merged 14 commits into
mainfrom
wc/update-deploy-invoke

Conversation

@wchigit
Copy link
Copy Markdown
Contributor

@wchigit wchigit commented Apr 16, 2026

What does this PR do?

Per #2245, MCP tools from deploy area are having problems being invoked. This PR update the tool description as well as the E2E test prompts.
deploy_app_logs_get tool is updated to a tool with LocalRequired=true because it reads local azure.yaml.

GitHub issue number?

[Link to the GitHub issue this PR addresses]

Pre-merge Checklist

  • Required for All PRs
    • Read contribution guidelines
    • PR title clearly describes the change
    • Commit history is clean with descriptive messages (cleanup guide)
    • Added comprehensive tests for new/modified functionality
    • Created a changelog entry if the change falls among the following: new feature, bug fix, UI/UX update, breaking change, or updated dependencies. Follow the changelog entry guide
  • For MCP tool changes:
    • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
    • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation
    • Validate README.md changes running the script ./eng/scripts/Process-PackageReadMe.ps1. See Package README
    • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
    • For tools with new names, including new tools or renamed tools, update consolidated-tools.json
    • For renamed tools, follow the Tool Rename Checklist and tag the PR with the breaking-change label
    • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description
  • Extra steps for Azure MCP Server tool changes:
    • Updated command list in servers/Azure.Mcp.Server/docs/azmcp-commands.md
    • Ran ./eng/scripts/Update-AzCommandsMetadata.ps1 to update tool metadata in azmcp-commands.md (required for CI)
    • Updated test prompts in servers/Azure.Mcp.Server/docs/e2eTestPrompts.md
    • 👉 For Community (non-Microsoft team member) PRs:
      • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
      • Manual tests run: added comment /azp run mcp - pullrequest - live to run Live Test Pipeline

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the Azure Deploy tool’s command/option descriptions (and related E2E prompts) to improve tool invocation for deploy-area scenarios in Namespace and All modes (per #2245), especially around CI/CD pipeline guidance.

Changes:

  • Relaxed several deploy option requirements and added defaults to reduce “missing required args” friction during tool selection/invocation.
  • Updated deploy command/group titles and descriptions to better steer the model toward deploy tools for deployment plans, pipeline guidance, IaC rules, diagrams, and logs.
  • Updated consolidated deploy tool metadata and E2E test prompts to better match the intended invocation phrasing.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
tools/Azure.Mcp.Tools.Deploy/src/Options/DeployOptionDefinitions.cs Makes several deploy options optional and sets defaults (pipeline + plan) to improve invocability.
tools/Azure.Mcp.Tools.Deploy/src/DeploySetup.cs Updates deploy area command group description to emphasize deploy tool usage for pipelines/plans/constraints.
tools/Azure.Mcp.Tools.Deploy/src/Commands/Plan/GetCommand.cs Refines deployment plan tool description for clearer “template/guidance” intent.
tools/Azure.Mcp.Tools.Deploy/src/Commands/Pipeline/GuidanceGetCommand.cs Updates title/description to better trigger CI/CD pipeline guidance invocation.
tools/Azure.Mcp.Tools.Deploy/src/Commands/Infrastructure/RulesGetCommand.cs Fixes “IaC” naming and improves description clarity for rules output.
tools/Azure.Mcp.Tools.Deploy/src/Commands/Architecture/DiagramGenerateCommand.cs Updates architecture diagram tool description (but currently lists unsupported types—see comment).
tools/Azure.Mcp.Tools.Deploy/src/Commands/App/LogsGetCommand.cs Marks logs tool as LocalRequired and updates description accordingly.
servers/Azure.Mcp.Server/src/Resources/consolidated-tools.json Updates deploy consolidated description and adds diagram tool to mapped tool list.
servers/Azure.Mcp.Server/docs/e2eTestPrompts.md Updates deploy E2E prompts to better align with desired deploy tool invocation.

Comment thread tools/Azure.Mcp.Tools.Deploy/src/Commands/Architecture/DiagramGenerateCommand.cs Outdated
wchigit and others added 3 commits April 16, 2026 15:41
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is red across 7/8 build platforms (only macos_arm64 passed) - that needs investigation before this can merge.

The option relaxation approach (Required=false with existing defaults) is the right fix for invocability. The title corrections (IaC, CI/CD) and LogsGetCommand LocalRequired=true are good. Diagram description now matches actual AzureServiceConstants enums after applying the Copilot suggestion.

One concern on the command group description in DeploySetup.cs - it lists the same capabilities three times in slightly different wording. This keyword-stuffing can hurt tool selection by diluting signal-to-noise ratio. A tighter single-pass description would serve the model better.

Comment thread tools/Azure.Mcp.Tools.Deploy/src/DeploySetup.cs Outdated
Copy link
Copy Markdown
Contributor

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few more items now that the description rewrites look good:

  1. servers/Azure.Mcp.Server/docs/azmcp-commands.md is stale. The metadata block for deploy app logs get still shows LocalRequired as false but this PR flips it to true, and several deploy plan get options (--target-app-service, --provisioning-tool, --source-type, --deploy-option) are shown as required even though they now have defaults. Run ./eng/scripts/Update-AzCommandsMetadata.ps1 per the PR checklist - this is the most likely cause of red CI.

  2. No changelog entry. The LocalRequired=true flip on deploy_app_logs_get will filter the tool out of HTTP/remote MCP mode (CommandFactoryToolLoader.cs:78, NamespaceToolLoader.cs:386). Combined with the Required-to-optional changes, that's user-observable behavior worth documenting per docs/changelog-entries.md.

  3. The LocalRequired flip is the right call - the command reads azure.yaml from the workspace - but worth noting in the PR description so remote-server users aren't surprised when the tool disappears.

  4. Did you run ToolDescriptionEvaluator for the new descriptions? The checklist asks for a confidence score of at least 0.4 and a top-3 ranking on each related test prompt.

The Copilot bot's diagram-description comment is stale - your current text matches AzureComputeServiceType exactly. Safe to resolve. The repetition I called out in DeploySetup.cs is also resolved in 43457c2.

Copy link
Copy Markdown
Contributor

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DeploySetup.cs description is much tighter now and the diagram description matches the actual AzureComputeServiceType / AzureServiceType enums. Both prior comments are addressed.

CI is still red on 4 of 8 build platforms on the latest commit (linux_arm64, linux_x64, macos_x64, windows_x64) - all mcp - pullrequest Azure DevOps build jobs. The macos_arm64, windows_arm64, and both musl docker builds pass. Worth investigating before merge - the failures look platform-specific rather than related to the description text changes, but the build logs will confirm.

@wchigit
Copy link
Copy Markdown
Contributor Author

wchigit commented Apr 22, 2026

The ToolDescriptionEvaluator is as below and passes the criterion:
image

@wchigit wchigit requested a review from a team as a code owner April 22, 2026 07:43
@wchigit
Copy link
Copy Markdown
Contributor Author

wchigit commented Apr 23, 2026

@jongio Thanks for the review. The pipeline failed because the LocalRequired value is not consistent anymore among all the tools in deploy area anymore. Could you help review this PR again? THX!

@wchigit wchigit requested a review from jongio April 24, 2026 05:23
Copy link
Copy Markdown
Contributor

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addresses prior feedback. The split of deploy_app_logs_get into its own namespace tool with localRequired: true is the right fix - it can't share the namespace with tools that don't need local access. Relaxing required options to use their existing defaults reduces invocation friction without changing behavior.

Comment thread tools/Azure.Mcp.Tools.Deploy/src/Commands/Pipeline/GuidanceGetCommand.cs Outdated
Co-authored-by: Copilot <copilot@github.com>
@wchigit wchigit enabled auto-merge (squash) April 27, 2026 02:15
Copy link
Copy Markdown
Contributor

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addresses prior feedback. The em-dash fix in GuidanceGetCommand.cs is a clean unicode cleanup - no functional change. CI is green across all 8 platforms. Good to merge.

@github-project-automation github-project-automation Bot moved this from Untriaged to In Progress in Azure MCP Server Apr 27, 2026
@wchigit wchigit merged commit fb593da into main Apr 27, 2026
15 checks passed
@wchigit wchigit deleted the wc/update-deploy-invoke branch April 27, 2026 13:20
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Azure MCP Server Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants