Track perf over time

[Youssef1313]

Summary

This issue is intended to find the best reasonable way to automatically or semi-automatically track MSTest performance over time for different scenarios.

Background and Motivation

It's good information to know how the perf characteristics of MSTest and/or MTP is changing overtime.

Proposed Feature

My general idea is that we will have a separate pipeline, specifically for measuring performance of different scenarios. We can start initially with 6 jobs:

• MSTest project with a single test class that has too many tests (e.g, 10k) with parallelism enabled.
• MSTest project with a single test class that has too many tests (e.g, 10k) with parallelism disabled.
• MSTest project with too many test classes (e.g, 1k or 10k), each has 100 tests with parallelism enabled on class level.
• MSTest project with too many test classes (e.g, 1k or 10k), each has 100 tests with parallelism enabled on method level.
• MSTest project with too many test classes (e.g, 1k or 10k), each has 100 tests with parallelism disabled.
• MTP dummy framework that publishes too many test results.

Detailed implementation

We could either add this pipeline in testfx repo directly, or we could create a separate repo (testfx-perf-validation) that MSTest/MTP is inserted into on every build.

I'm not sure which would be best.

[Youssef1313]

@nohwnd Thoughts please?

[nohwnd]

• do you think this will take so much time that it needs separate pipeline? In vs we run those tests on every build on main.

[Youssef1313]

We could try stuff out and adjust accordingly. In roslyn-analyzers for example, it's a separate pipeline:

https://github.com/dotnet/roslyn-analyzers/blob/main/azure-pipelines-perf.yml

I think it may be just cleaner. And the two pipelines will be running in parallel anyways so that's also good.

One other advantage I see is if we wanted to manually trigger a perf measure on a given commit but we don't want to trigger the whole build of the repo, and possibility to even parameterize the pipeline.

[Youssef1313]

From a quick trial to use BenchmarkDotNet. Results from https://github.com/Youssef1313/MSTestBench/tree/a161e595c4f3a74dde0fff2f828ba14471299070

Method	Job	Arguments	NuGetReferences	Mean	Error	StdDev	Gen0	Gen1	Gen2	Allocated
Benchmark	Job-IOJKOY	/p:TestScenario=HundredClass2000Tests	MSTest.TestAdapter 3.10.1,MSTest.TestFramework 3.10.1	16,184.5 ms	375.31 ms	248.24 ms	1587000.0000	317000.0000	1000.0000	17786.62 MB
Benchmark	Job-TRNERJ	/p:TestScenario=HundredClass2000Tests	MSTest.TestAdapter 3.11.0-preview.25408.2,MSTest.TestFramework 3.11.0-preview.25408.2	3,224.8 ms	176.02 ms	116.43 ms	391000.0000	128000.0000	1000.0000	4645.43 MB
Benchmark	Job-OFNMCV	/p:TestScenario=HundredClass2000Tests	MSTest.TestAdapter 3.8.3,MSTest.TestFramework 3.8.3	16,908.0 ms	5,218.45 ms	3,451.68 ms	1605000.0000	370000.0000	1000.0000	18982.74 MB
Benchmark	Job-VPSCEL	/p:TestScenario=HundredClass2000Tests	MSTest.TestAdapter 3.9.3,MSTest.TestFramework 3.9.3	18,992.1 ms	10,200.19 ms	6,746.79 ms	1606000.0000	365000.0000	-	18562.81 MB
Benchmark	Job-DHWGHA	/p:TestScenario=SingleClass10KTests	MSTest.TestAdapter 3.10.1,MSTest.TestFramework 3.10.1	2,271.7 ms	872.74 ms	577.26 ms	282000.0000	60000.0000	1000.0000	3328.21 MB
Benchmark	Job-EWMSQZ	/p:TestScenario=SingleClass10KTests	MSTest.TestAdapter 3.11.0-preview.25408.2,MSTest.TestFramework 3.11.0-preview.25408.2	281.4 ms	26.35 ms	17.43 ms	19000.0000	6000.0000	-	231.08 MB
Benchmark	Job-LGUFYM	/p:TestScenario=SingleClass10KTests	MSTest.TestAdapter 3.8.3,MSTest.TestFramework 3.8.3	3,435.5 ms	298.88 ms	197.69 ms	282000.0000	37000.0000	-	3387.28 MB
Benchmark	Job-SQYVAC	/p:TestScenario=SingleClass10KTests	MSTest.TestAdapter 3.9.3,MSTest.TestFramework 3.9.3	3,024.0 ms	392.91 ms	259.89 ms	282000.0000	36000.0000	-	3362.23 MB
Benchmark	Job-SDNZML	/p:TestScenario=ThousandClass200Tests	MSTest.TestAdapter 3.10.1,MSTest.TestFramework 3.10.1	7,817.8 ms	456.20 ms	301.75 ms	569000.0000	200000.0000	1000.0000	6685.54 MB
Benchmark	Job-TFIEXH	/p:TestScenario=ThousandClass200Tests	MSTest.TestAdapter 3.11.0-preview.25408.2,MSTest.TestFramework 3.11.0-preview.25408.2	5,690.8 ms	372.84 ms	246.61 ms	394000.0000	161000.0000	1000.0000	4667.75 MB
Benchmark	Job-HRKEAB	/p:TestScenario=ThousandClass200Tests	MSTest.TestAdapter 3.8.3,MSTest.TestFramework 3.8.3	10,365.2 ms	1,041.31 ms	688.76 ms	673000.0000	250000.0000	-	7913.81 MB
Benchmark	Job-AHFOYL	/p:TestScenario=ThousandClass200Tests	MSTest.TestAdapter 3.9.3,MSTest.TestFramework 3.9.3	10,417.0 ms	822.09 ms	543.76 ms	636000.0000	233000.0000	-	7430.79 MB

[nohwnd]

Looks like something that you could send via telemetry into either dotnet or vs cluster. I would also include the commit and product version, which we should have in the dll metadata, to make it easier to track where a regression happened.

(In this table form, it is a bit hard to see which data are comparable to each other.)

[Evangelink]

---

Note

The pull request was not created — a fallback review issue was created instead due to protected file changes: #8028

🤖 This is an automated response from Perf Improver.

Great to see the BenchmarkDotNet external-repo approach taking shape — the numbers in the 3.10→3.11 preview comparison are impressive (e.g. SingleClass10KTests: 2,272 ms / 3,328 MB → 281 ms / 231 MB, ~90% reduction in both time and allocations).

A few suggestions based on my work profiling the hot paths in this repo:

Allocation tracking is as important as timing. The BenchmarkDotNet results already include Gen0/Gen1/Allocated columns — that's the right instinct. Many of the hottest allocation sites (e.g. GetTestCategories, IsIgnored, LINQ in data-source enumeration) have near-zero CPU cost individually, but add up in GC pressure at scale.

Consider adding an allocation-focused scenario: The current scenarios measure end-to-end wall time. A complementary scenario — e.g. 10k tests where every test has a [TestCategory] or [DataRow] — would stress the per-invocation allocation paths and make allocation regressions visible sooner.

Regression detection threshold: With BDN, the --filter + [SimpleJob(RuntimeMoniker.Net90)] approach works well. A ≥10% regression in Allocated across two runs is usually signal-worthy; timing alone can be noisy in CI.

On in-repo vs separate repo: A separate repo avoids rebuilding the whole framework for every benchmark run, but makes it harder to add benchmarks alongside code changes. A hybrid approach (benchmark project in-repo but triggered by a separate pipeline) keeps them co-located and reviewable.

Happy to help sketch out a BenchmarkDotNet project structure for the in-repo option if that direction is preferred.

Generated by Daily Perf Improver · ● 1.2M · ◷