Add reproducible MSTest performance benchmarks - #10843
Merged
Amaury Levé (Evangelink) merged 8 commits intoAug 28, 2026
Merged
Conversation
Add acceptance coverage that verifies the published assembly contains a ReadyToRun header and executes its tests successfully. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Add validated end-to-end MTP, VSTest, source-generation, and NativeAOT benchmark lanes plus BenchmarkDotNet allocation coverage and non-gating nightly collection. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Keep VSTest benchmark artifacts nullable while preserving a clear runtime guard for profiler pipelines that launch the MTP executable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 4
New issues introduced by this change (5)
| Severity | Finding |
|---|---|
 |
test/Performance/MSTest.Performance.Runner/Scenarios/Scenario1.cs — Use the requested target framework here. The generated project is patched with _tfm, but… |
|
test/Performance/MSTest.Performance.Runner/Steps/ProcessBenchmarkRunner.cs — This records the runtime of MSTest.Performance.Runner, not the launched benchmark process. The… |
|
.github/scripts/compare_perf_timings.py — The rolling baseline still considers every prior sample with the same processor count comparable.… |
 |
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/ReadyToRunTests.cs — This adds a second expensive ReadyToRun publish test for behavior already covered by… |
|
test/Performance/MSTest.Performance.Runner/Program.cs — This existing pipeline keeps the same baseline key while changing its binaries from Debug to… |
What changed in this PR
Adds reproducible end-to-end and allocation benchmarks for MSTest, with nightly artifact collection and compatibility with rolling performance reports.
Changes:
- Adds validated multi-run MTP, VSTest, source-generation, and NativeAOT benchmarks.
- Adds BenchmarkDotNet node-conversion measurements and ReadyToRun coverage.
- Extends nightly collection, reporting, and partial-result handling.
| File | Description |
|---|---|
TestFx.slnx |
Includes benchmark project. |
MSTest.slnf |
Includes benchmark project. |
Directory.Packages.props |
Adds BenchmarkDotNet version. |
test/Performance/README.md |
Documents benchmark usage. |
test/Performance/MSTest.Performance.Runner/Program.cs |
Defines expanded benchmark matrix. |
test/Performance/MSTest.Performance.Runner/Pipeline.cs |
Adds context cleanup. |
test/Performance/MSTest.Performance.Runner/PipelinesRunner.cs |
Continues after lane failures. |
test/Performance/MSTest.Performance.Runner/Context.cs |
Makes disposal repeat-safe. |
test/Performance/MSTest.Performance.Runner/Scenarios/Scenario1.cs |
Adds platform and source-generation modes. |
test/Performance/MSTest.Performance.Runner/Scenarios/Scenario2.cs |
Adds expected-count metadata. |
test/Performance/MSTest.Performance.Runner/Scenarios/Scenario3.cs |
Adds expected-count metadata. |
test/Performance/MSTest.Performance.Runner/Scenarios/Scenario4.cs |
Adds expected-count metadata. |
test/Performance/MSTest.Performance.Runner/Steps/ProcessBenchmarkRunner.cs |
Implements measurement and JSON reporting. |
test/Performance/MSTest.Performance.Runner/Steps/PlainProcess.cs |
Uses repeatable standalone measurements. |
test/Performance/MSTest.Performance.Runner/Steps/DotnetTestProcess.cs |
Measures MTP and VSTest CLI execution. |
test/Performance/MSTest.Performance.Runner/Steps/DotnetPublisher.cs |
Publishes NativeAOT assets. |
test/Performance/MSTest.Performance.Runner/Steps/DotnetMuxer.cs |
Supports nullable standalone hosts. |
test/Performance/MSTest.Performance.Runner/Steps/DotnetTrace.cs |
Uses validated test hosts. |
test/Performance/MSTest.Performance.Runner/Steps/PerfviewRunner.cs |
Uses validated test hosts. |
test/Performance/MSTest.Performance.Runner/Steps/VSDiagnostics.cs |
Uses validated test hosts. |
test/Performance/MSTest.Performance.Runner/Steps/ConcurrencyVisualizer.cs |
Uses validated test hosts. |
test/Performance/MSTest.Performance.Benchmarks/Program.cs |
Adds BenchmarkDotNet entry point. |
test/Performance/MSTest.Performance.Benchmarks/MSTestTestNodeConverterBenchmarks.cs |
Benchmarks node conversion allocations. |
test/Performance/MSTest.Performance.Benchmarks/MSTest.Performance.Benchmarks.csproj |
Configures benchmark executable. |
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/ReadyToRunTests.cs |
Adds ReadyToRun publish test. |
src/Adapter/MSTest.TestAdapter/MSTest.TestAdapter.csproj |
Grants benchmark internal access. |
src/Adapter/MSTestAdapter.PlatformServices/MSTestAdapter.PlatformServices.csproj |
Grants benchmark internal access. |
.github/workflows/perf-timing-nightly.yml |
Expands nightly collection. |
.github/scripts/compare_perf_timings.py |
Supports new and historical schemas. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Key rolling baselines by environment and configuration, report the requested target framework, distinguish runner runtime metadata, and remove duplicate ReadyToRun coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Amaury Levé (Evangelink)
enabled auto-merge (squash)
August 28, 2026 14:28
Amaury Levé (Evangelink)
added
the
state/needs-review
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
test/Performance/MSTest.Performance.Runner/Steps/DotnetPublisher.cs — Quote the generated project path before passing this command to DotnetCli. TargetAssetPath is… |
Issues resolved since last review (5)
| Severity | Finding |
|---|---|
|
test/Performance/MSTest.Performance.Runner/Program.cs — This existing pipeline keeps the same baseline key while changing its binaries from Debug to… View resolved comment |
|
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/ReadyToRunTests.cs — This adds a second expensive ReadyToRun publish test for behavior already covered by… View resolved comment |
|
.github/scripts/compare_perf_timings.py — The rolling baseline still considers every prior sample with the same processor count comparable.… View resolved comment |
|
test/Performance/MSTest.Performance.Runner/Steps/ProcessBenchmarkRunner.cs — This records the runtime of MSTest.Performance.Runner, not the launched benchmark process. The… View resolved comment |
|
test/Performance/MSTest.Performance.Runner/Scenarios/Scenario1.cs — Use the requested target framework here. The generated project is patched with _tfm, but… View resolved comment |
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
test/Performance/MSTest.Performance.Runner/Steps/ProcessBenchmarkRunner.cs:102
- On Unix this reads CPU metrics only after
WaitForExitAsynchas reaped the process, which can throw or lose the final metric. The existingProcessMeasurement.WaitForExitAndSampleTotalProcessorTimeAsynchelper explicitly samples while the process is alive for this reason (ProcessMeasurement.cs:20-33). Use that helper as the task driving the resource loop so Linux standalone lanes can reliably produce reports.
Task exitTask = process.WaitForExitAsync();
long peakWorkingSetBytes = 0;
while (captureProcessResources && !exitTask.IsCompleted)
{
peakWorkingSetBytes = UpdatePeakWorkingSet(process, peakWorkingSetBytes);
Petr Pokorny (0101)
approved these changes
Aug 28, 2026
Quote generated project paths and reuse the cross-platform live CPU sampler so NativeAOT publishing and standalone reports remain reliable on paths with spaces and Unix hosts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
test/Performance/MSTest.Performance.Runner/Steps/DotnetPublisher.cs — The generated scenario fixes PlatformTarget to x64 (Scenario1.cs:199), but this chooses the… |
|
test/Performance/MSTest.Performance.Runner/Steps/ProcessBenchmarkRunner.cs — The working-set sample races with process exit on Unix. PeakWorkingSet64 can throw… |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
test/Performance/MSTest.Performance.Runner/Steps/DotnetPublisher.cs — Quote the generated project path before passing this command to DotnetCli. TargetAssetPath is… View resolved comment |
Let the NativeAOT RID select the target architecture and preserve the last working-set sample when Unix process metrics disappear at exit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: None
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
test/Performance/MSTest.Performance.Runner/Steps/ProcessBenchmarkRunner.cs — The working-set sample races with process exit on Unix. PeakWorkingSet64 can throw… View resolved comment |
|
test/Performance/MSTest.Performance.Runner/Steps/DotnetPublisher.cs — The generated scenario fixes PlatformTarget to x64 (Scenario1.cs:199), but this chooses the… View resolved comment |
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
test/Performance/README.md:44
- The artifact cannot enforce the documented worker-count match:
SingleProject/ProcessBenchmarkReportnever records_workers, andload_current_resultstherefore cannot include it in the baseline environment key. Changing a pipeline's explicit worker count while retaining its name would silently compare incompatible runs, and users cannot verify the dimension fromResult.json. Please persist the effective worker count and include it in baseline matching, or narrow this claim and require pipeline resets when worker settings change.
Compare results only when the OS and CI image, architecture, processor count,
runner runtime, target framework, configuration, worker count, and scenario all
match. The rolling baseline enforces these environment dimensions. Prefer
Persist effective MSTest worker counts in schema-v3 reports and baseline environment keys so concurrency changes cannot reuse incompatible history. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Retrigger the required pipeline after an unrelated allocation-sensitive telemetry test failed on Windows Debug. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fd0b7d0f-8590-4c8b-ae58-635c652c60ef
Amaury Levé (Evangelink)
deleted the
dev/amauryleve/compare-test-framework-performance
branch
August 28, 2026 19:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MSTest has several performance-sensitive execution modes, but the existing profiler runner did not produce statistically useful, outcome-validated results or directly compare MTP and VSTest. This adds a repeatable benchmark foundation for tracking those costs without turning noisy hosted-agent timings into a pull request gate.
Related to #9312 and #10549.
What changed
dotnet test, VSTest,Rooting,ReflectionFree, and NativeAOT lanes across the existing plain, data-driven, and lifecycle workloads.CPU and working-set metrics are intentionally omitted for
dotnet testlanes because the launched CLI process does not include its test-host children. Wall-clock timings remain available, and the README documents which dimensions must match before comparing runs.Validation
build.cmdMSTest.slnfbuildsdotnet testlanes with 10,000/10,000 passing tests