Skip to content

Speed up CLI argument escaping tests - #1773

Merged
PranavSenthilnathan merged 1 commit into
mainfrom
pranavsenthilnathan-speed-up-cli-argument-test
Jul 29, 2026
Merged

Speed up CLI argument escaping tests#1773
PranavSenthilnathan merged 1 commit into
mainfrom
pranavsenthilnathan-speed-up-cli-argument-test

Conversation

@PranavSenthilnathan

Copy link
Copy Markdown
Contributor

Replace repeated MCP handshake/tool calls with a lightweight TestServer mode that echoes the parsed CLI argument through stderr and exits cleanly. This preserves real child-process coverage across Windows cmd.exe, Unix dotnet, and Mono launch paths.

Targeted runtime across all frameworks dropped from 3m 53s to ~8s (~30× faster), saving approximately 3½ minutes per affected CI leg and potentially from overall PR validation time.

Validation: dotnet build and all 38 targeted cases pass on net10.0, net9.0, net8.0, and net472.

Generated with GitHub Copilot.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 323a2062-9493-4be4-a710-f6c2b9e15a49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve

The changes are isolated to test infrastructure and make the escaping tests faster while maintaining child-process coverage and adding explicit exit-code validation.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR accelerates the StdioClientTransport CLI argument escaping test suite by avoiding full MCP handshake/tool invocations and instead using a lightweight TestServer execution mode that echoes the parsed CLI argument to stderr and exits immediately, preserving real child-process invocation coverage across Windows cmd.exe, Unix dotnet, and Mono launch paths.

Changes:

  • Added a --echo-cli-arg-and-exit mode to the TestServer that writes CLI_ARG:<json> to stderr and exits.
  • Updated EscapesCliArgumentsCorrectly to capture the echoed value from stderr, JSON-parse it, and assert a clean process exit (exit code 0) via completion details.
File summaries
File Description
tests/ModelContextProtocol.TestServer/Program.cs Adds an early-exit mode that echoes the parsed --cli-arg= value to stderr for fast, deterministic test verification.
tests/ModelContextProtocol.Tests/Transport/StdioClientTransportTests.cs Reworks the escaping theory test to use the new echo mode + stderr capture, and asserts clean transport/process shutdown details.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@tarekgh tarekgh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Further optimization opportunities

  1. Biggest remaining cost = 38 separate child-process launches (~250ms each ≈ 9-10s per framework). To go materially faster, collapse the  [Theory]  into a single  [Fact]  that launches the child once, passing all 38 values ( --cli-arg=  repeated), with the echo mode emitting one  CLI_ARG::  line per value. That turns 38 launches into 1 per framework (sub-second). Tradeoff: you lose per-case failure granularity, so mitigate by collecting all mismatches and reporting them together ( Assert.All  over the parsed results) rather than failing on the first. The escaping logic is still fully exercised because each argument is independently escaped by the transport's argument-list builder.
  2. Trim the modern-TFM matrix for this theory. The escaping code differs between net472 (legacy argument-string building) and modern .NET ( ArgumentList ), so net472 + one modern TFM are the meaningful axes. net8.0/net9.0/net10.0 almost certainly run identical escaping code, so running the full 38-case theory on all three is largely redundant. Restricting the heavy theory to net-latest + net472 (leaving a smoke case on the others) roughly halves the remaining cost and is orthogonal to no. 1.

@PranavSenthilnathan

Copy link
Copy Markdown
Contributor Author

Further optimization opportunities

  1. Biggest remaining cost = 38 separate child-process launches (~250ms each ≈ 9-10s per framework). To go materially faster, collapse the  [Theory]  into a single  [Fact]  that launches the child once, passing all 38 values ( --cli-arg=  repeated), with the echo mode emitting one  CLI_ARG::  line per value. That turns 38 launches into 1 per framework (sub-second). Tradeoff: you lose per-case failure granularity, so mitigate by collecting all mismatches and reporting them together ( Assert.All  over the parsed results) rather than failing on the first. The escaping logic is still fully exercised because each argument is independently escaped by the transport's argument-list builder.

I wanted to keep the cases separate, but I don't feel strongly about it. We can always do this in a future PR.

  1. Trim the modern-TFM matrix for this theory. The escaping code differs between net472 (legacy argument-string building) and modern .NET ( ArgumentList ), so net472 + one modern TFM are the meaningful axes. net8.0/net9.0/net10.0 almost certainly run identical escaping code, so running the full 38-case theory on all three is largely redundant. Restricting the heavy theory to net-latest + net472 (leaving a smoke case on the others) roughly halves the remaining cost and is orthogonal to no. 1.

This warrants a more holistic discussion about what tests to condition and on which TFMs. This will give the biggest perf improvement but it's at the cost of losing coverage. I think we should get the low hanging perf wins first and then consider this.

@PranavSenthilnathan
PranavSenthilnathan merged commit 155f619 into main Jul 29, 2026
17 of 18 checks passed
@PranavSenthilnathan
PranavSenthilnathan deleted the pranavsenthilnathan-speed-up-cli-argument-test branch July 29, 2026 21:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-tests Issue related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants