Skip to content

Confidence score improvements for command selection#92

Merged
nishtha489 merged 25 commits intomainfrom
users/nishtha/confidence-improvements
Oct 6, 2025
Merged

Confidence score improvements for command selection#92
nishtha489 merged 25 commits intomainfrom
users/nishtha/confidence-improvements

Conversation

@nishtha489
Copy link
Copy Markdown
Contributor

@nishtha489 nishtha489 commented Aug 29, 2025

What does this PR do?

The PR includes improvements in the commands description to help improve the confidence score while model selection.

GitHub issue number?

Associated issue - https://github.com/Azure/azure-mcp/issues/832

Result scores for confidence

Prompt: Create a basic URL test using the following endpoint URL that runs for 30 minutes with 45 virtual users. The test name is with the test id and the load testing resource is in the resource group in my subscription
Expected tool: azmcp_loadtesting_test_create
0.585388 azmcp_loadtesting_test_create *** EXPECTED ***
0.531331 azmcp_loadtesting_testresource_create
0.508690 azmcp_loadtesting_testrun_create

Prompt: Get the load test with id in the load test resource in resource group
Expected tool: azmcp_loadtesting_test_get
0.642258 azmcp_loadtesting_test_get *** EXPECTED ***
0.608693 azmcp_loadtesting_testresource_list
0.574354 azmcp_loadtesting_testresource_create

Prompt: Create a load test resource in the resource group in my subscription
Expected tool: azmcp_loadtesting_testresource_create
0.717674 azmcp_loadtesting_testresource_create *** EXPECTED ***
0.596680 azmcp_loadtesting_testresource_list
0.514720 azmcp_loadtesting_test_create

Prompt: List all load testing resources in the resource group in my subscription
Expected tool: azmcp_loadtesting_testresource_list
0.738027 azmcp_loadtesting_testresource_list *** EXPECTED ***
0.591857 azmcp_loadtesting_testresource_create
0.577408 azmcp_group_list

Prompt: Create a test run using the id for test in the load testing resource in resource group . Use the name of test run and description as
Expected tool: azmcp_loadtesting_testrun_create
0.621803 azmcp_loadtesting_testrun_create *** EXPECTED ***
0.592748 azmcp_loadtesting_testresource_create
0.540789 azmcp_loadtesting_test_create

Prompt: Get the load test run with id in the load test resource in resource group
Expected tool: azmcp_loadtesting_testrun_get
0.625461 azmcp_loadtesting_test_get
0.603773 azmcp_loadtesting_testrun_get *** EXPECTED ***
0.568474 azmcp_loadtesting_testresource_list

Prompt: Get all the load test runs for the test with id in the load test resource in resource group
Expected tool: azmcp_loadtesting_testrun_list
0.615977 azmcp_loadtesting_testrun_list *** EXPECTED ***
0.606058 azmcp_loadtesting_test_get
0.569145 azmcp_loadtesting_testrun_get

Prompt: Update a test run display name as for the id for test in the load testing resource in resource group .
Expected tool: azmcp_loadtesting_testrun_update
0.659812 azmcp_loadtesting_testrun_update *** EXPECTED ***
0.509199 azmcp_loadtesting_testrun_create
0.454745 azmcp_loadtesting_testrun_get

Pre-merge Checklist

  • Required for All PRs
    • Read contribution guidelines
    • PR title clearly describes the change
    • Commit history is clean with descriptive messages (cleanup guide)
    • Added comprehensive tests for new/modified functionality
    • Updated CHANGELOG.md for product changes (features, bug fixes, UI/UX, updated dependencies)
    • Spelling check passes: .\eng\common\spelling\Invoke-Cspell.ps1
  • For MCP tool changes:
    • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
    • Updated README.md documentation
    • Updated command list in /docs/azmcp-commands.md
    • Updated test prompts in /docs/e2eTestPrompts.md
    • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
  • 👉 For Community (non-Azure team member) PRs:
    • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
    • Manual tests run: added comment /azp run azure - mcp to run Live Test Pipeline

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves command descriptions in the Azure Load Testing MCP tool to enhance model confidence scores during command selection. The changes focus on making command descriptions more specific and detailed to help the model better distinguish between similar operations.

  • Updated the main Load Testing service description with detailed capability explanations and usage guidance
  • Enhanced individual command descriptions with clearer distinctions between test creation, test run management, and resource operations
  • Added explicit examples and parameter information to reduce ambiguity between related commands

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
LoadTestingSetup.cs Updated main service description with comprehensive details about Load Testing capabilities and usage scenarios
TestRunUpdateCommand.cs Added clarification that this only updates test run metadata, not test configuration or resources
TestRunListCommand.cs Enhanced description to clearly specify this lists test runs for a given test ID with examples
TestRunGetCommand.cs Clarified this retrieves details for a specific test run ID, not test configuration
TestRunCreateCommand.cs Simplified description to emphasize this only creates test runs for existing tests
TestResourceCreateCommand.cs Clarified this only creates Azure resources, not test plans or runs
TestGetCommand.cs Enhanced description to distinguish between test configuration retrieval vs test run data
TestCreateCommand.cs Added detailed explanation with examples to clarify this creates test configurations, not runs or resources

Comment thread tools/Azure.Mcp.Tools.LoadTesting/src/LoadTestingSetup.cs Outdated
Comment thread tools/Azure.Mcp.Tools.LoadTesting/src/Commands/LoadTest/TestCreateCommand.cs Outdated
nishtha489 and others added 2 commits August 29, 2025 22:25
…tRunUpdateCommand.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…eateCommand.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@nishtha489 nishtha489 enabled auto-merge (squash) September 1, 2025 14:54
Copy link
Copy Markdown
Contributor

@g2vinay g2vinay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a Changelog entry for the change, rest looks fine

@nishtha489 nishtha489 requested a review from a team as a code owner September 5, 2025 19:50
Comment thread eng/vscode/CHANGELOG.md Outdated
Comment thread tools/Azure.Mcp.Tools.LoadTesting/src/Commands/LoadTest/TestGetCommand.cs Outdated
Comment thread tools/Azure.Mcp.Tools.LoadTesting/src/Commands/LoadTest/TestCreateCommand.cs Outdated
feiskyer pushed a commit to feiskyer/microsoft-mcp that referenced this pull request Sep 8, 2025
@nishtha489 nishtha489 requested a review from jongio September 18, 2025 14:30
Nishtha . added 2 commits September 23, 2025 18:35
…users/nishtha/confidence-improvements"

This reverts commit e03a3df, reversing
changes made to 1766421.
@g2vinay
Copy link
Copy Markdown
Contributor

g2vinay commented Sep 23, 2025

@jongio requested changes, lets wait on his approval.

@github-project-automation github-project-automation Bot moved this from Untriaged to In Progress in Azure MCP Server Oct 2, 2025
Copy link
Copy Markdown
Member

@xiaomi7732 xiaomi7732 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@nishtha489 nishtha489 merged commit d9114e5 into main Oct 6, 2025
26 checks passed
@nishtha489 nishtha489 deleted the users/nishtha/confidence-improvements branch October 6, 2025 17:10
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Azure MCP Server Oct 6, 2025
colbytimm pushed a commit to colbytimm/microsoft-mcp that referenced this pull request Dec 8, 2025
* confidence improvements

* Update tools/Azure.Mcp.Tools.LoadTesting/src/Commands/LoadTestRun/TestRunUpdateCommand.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tools/Azure.Mcp.Tools.LoadTesting/src/Commands/LoadTest/TestCreateCommand.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Nishtha . <nishtha@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants