Azure Devops Pipeline with a new MSTest runner 3.2.0 #2175
Comments
|
Hi @SymbioticKilla, MSTest runner has a compatibility layer with the past so you could run your pipeline as-is (you won't get the benefit of the new features/perfs...). If you want to take full advantage of the runner, you should be following this doc page for enabling dotnet test with the runner. For the code coverage, if you want to use coverlet (as there is currently no specific adapter), you will have to use the global tool. Please refer to coverlet doc page https://github.com/coverlet-coverage/coverlet#net-global-tool-guide-suffers-from-possible-known-issue. If you don't mind using Microsoft Code Coverage (which we recommend - check out this article for some of the reasons https://devblogs.microsoft.com/dotnet/whats-new-in-our-code-coverage-tooling/?utm_content=buffer9eae6&utm_medium=social&utm_source=linkedin.com&utm_campaign=buffer), then you can use the following configuration. Project <PropertyGroup>
<EnableMSTestRunner>true</EnableMSTestRunner>
<OutputType>Exe</OutputType>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="1.0.0" />
<PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="17.10.1" />
<PackageReference Include="MSTest" Version="3.2.0" />
</ItemGroup>
</PropertyGroup>My pipeline: - task: DotNetCoreCLI@2
displayName: Build solution
inputs:
command: "build"
projects: '**/Solution.sln'
arguments: '-c Release'
- task: DotNetCoreCLI@2
displayName: Run unit tests
continueOnError: true
inputs:
command: 'test'
projects: '**/Solution.sln'
arguments: '-c Release -p:TestingPlatformCommandLineArguments=" --results-directory \"$(Build.SourcesDirectory)/TestResults/Coverage/\" --report-trx --coverage --coverage-output-format covertura "'
publishTestResults: false
- task: reportgenerator@5
displayName: Generate code coverage report
inputs:
reports: '$(Build.SourcesDirectory)/TestResults/Coverage/*/coverage.cobertura.xml'
targetdir: '$(Build.SourcesDirectory)/TestResults/CoverageReport'
- task: PublishCodeCoverageResults@1
displayName: Publish code coverage results
inputs:
codeCoverageTool: "Cobertura"
summaryFileLocation: "$(Build.SourcesDirectory)/TestResults/CoverageReport/Cobertura.xml"
reportDirectory: "$(Build.SourcesDirectory)/TestResults/CoverageReport"
- task: PublishTestResults@2
displayName: Publish test results
inputs:
testResultsFormat: 'VSTest'
testResultsFiles: '**/*.trx'
searchFolder: '$(Build.SourcesDirectory)/TestResults/Coverage'
mergeTestResults: true
|
Evangelink
added
Type: Discussion
Needs: Author Feedback
Area: Testing Platform
|
@Evangelink I have changed couple of things in your proposal for pipeline: --coverage-output-format cobertura 2.) reportgenerator@5 => Structure for code coverage output is different. The reports folder and name are changed reports: '$(Build.SourcesDirectory)/TestResults/Coverage/*.cobertura.xml'3.) Disable autogenerate in PublishCodeCoverageResults@1 env:
DISABLE_COVERAGE_AUTOGENERATE: 'true'I have couple of questions: Thanks! |
microsoft-github-policy-service
bot
added
Needs: Attention 👋
and removed
Needs: Author Feedback
labels
|
Sorry for the missed edits and typos, I wrote by head on GH so didn't get much safetynet :( For code coverage improvements, @jakubch1 will give you some details.
You should have it all (assuming you updated the test projects as stated in the guide). There is one extra thing, if you are not using
You can add the
It depends how big you want, but you can refer to our samples of performance diff between VSTest and MSTest runner (https://github.com/microsoft/testfx/tree/main/samples/runner_vs_vstest#mstest-runner-vs-vstest) |
|
@SymbioticKilla for code coverage please check those examples: https://github.com/microsoft/codecoverage/tree/main/samples/Algorithms |
|
Thank you! I have tried to check performance. It is weird... With a new MSTest Runner the tests are slower. 1.) Test step with a new MS Test Runner in Azure Devops: /_work/_tool/dotnet/dotnet test /_work/11/s/Source/Solution.sln --no-restore --no-build -c Release -p:TestingPlatformCommandLineArguments= --results-directory "/_work/11/s/TestResults/Coverage/" --report-trx --coverage --coverage-output-format coberturaExecution time: ~12 seconds 2.) Test step with an old VSTest /_work/_tool/dotnet/dotnet test /_work/18/s/Source/Solution.sln --no-restore --no-build -c Release --results-directory /_work/18/s/TestResults/Coverage/ --collect XPlat Code Coverage --logger trxExecution time: ~6 Seconds New tests .csproj: <PropertyGroup>
<EnableMSTestRunner>true</EnableMSTestRunner>
<OutputType>Exe</OutputType>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="1.0.0" />
<PackageReference Include="MSTest" Version="3.2.0" />
<PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="17.10.1" />
<PackageReference Include="StyleCop.Analyzers" Version="1.2.0-beta.507">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="System.Linq.Async" Version="6.0.1" />
</ItemGroup> |
|
Could you please share a bit more detail about your solution? (Unless it is opensource and you can just link me to your repo :)) I would like to investigate this, but we only heard about one similar issue that is using a custom ITestDataSource, and there we still don't know the cause because it is huge and super hard to build.
You could also create a VS feedback ticket (https://developercommunity.visualstudio.com/home) and privately attach any logs that you can collect, or a subset of your solution that reproes the issue. collecting logs of runner: https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-runner-extensions#built-in-options collecting logs of VSTEST: https://github.com/microsoft/vstest/blob/main/docs/diagnose.md |
|
@nohwnd I think he just setup some CI using the perf samples we have in the repo:
|
|
I've tried this on my local windows system, and I am seeing 6 seconds on my trivial test solution with 2 projects and 2 tests (in total), when coverlet is enabled. Are you running the old command against the same solution as the new one? The new solution does not have coverlet installed in the project, and so the whole overhead of coverage is skipped. My proj for comparison <Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>
<PropertyGroup>
<EnableMSTestRunner>true</EnableMSTestRunner>
<OutputType>Exe</OutputType>
<TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="1.0.0" />
<PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="17.10.1" Condition=" $(TestingPlatformDotnetTestSupport) == 'true'" />
<PackageReference Include="MSTest" Version="3.2.0" />
<PackageReference Include="coverlet.collector" Version="1.0.1" Condition=" $(TestingPlatformDotnetTestSupport) == 'false'" />
</ItemGroup>
</Project>Old + coverlet: ~7s Old + coverlet: Old without coverlet new with coverage: new without coverage |
|
Thanks for testing! |
|
Thanks, it would be helpful if you shared sanitized output of the build, if possible. |
|
@SymbioticKilla when doing |
|
Hey! It is something with our solution. It is more complicated, so there could be some hidden problems. Both in build pipeline and locally I'm getting worse performance. I could minimize my search. I have tried to reduce to minimum: I took test project and deleted all tests and left just one with no logic. The time spent was still high (almost the same), so I have removed the reference to main project and time was reduced from 5+ seconds to 2+ seconds. So just reference to a project reduces the test time. Can you give a hint? |
microsoft-github-policy-service
bot
added
Needs: Attention 👋
and removed
Needs: Author Feedback
labels
Great news :)
We have received another report that one of our customer has one (only once it seems) project that is 2-3 times slower with the runner than it used to be with VSTest. We are trying to investigate the problem as we have access to the binaries. Hopefully that would be a scenario similar to yours. BTW do you happen to have some other tech/languages mixed in this project (C, C++, JS...)? |
|
Pure .NET 7 :) I will try to reproduce it in a new project for you till end of this week. |
Ok so that's not something in common with the other project :)
That would be huge! |
|
@SymbioticKilla We found the issue. We are hooking up to the VSTest target, which dotnet/sdk allows us to replace. In the original VStest task we conditionally build the tested project (to allow dotnet build -t:vstest, to work). In our new target we copied this approach, but forgot this condition, so every project will "build" again. "build" in quotes because it invokes msbuild, that does up-to-date check and returns, but it still takes a second or more per project. We will release a fix for this soon. |
|
Glad to hear! Waiting to test 👍 |
|
Update: This is full guide to accessing our preview packages: https://aka.ms/mstest/preview @SymbioticKilla would you be interested in testing the fix before its release? If so, you could use the following feed https://github.com/microsoft/testfx/blob/main/NuGet.config#L8. I'll get back to you with the min version that contains the fix. |
|
Closing as done. We will reopen if there is still some issue for you. |
|
@SymbioticKilla MSTest 3.2.1 was just released |
|
@Evangelink On windows (locally) is not reproducible. 1.0.0 works as expected. Should I create a new bug report? |
Love it!
Oh sorry! Yes please do! Could you try to provide some extra info about the project? Can you for example send me (by email or through developer community - please follow that link for how to upload private info https://learn.microsoft.com/cpp/overview/how-to-report-a-problem-with-the-visual-cpp-toolset?view=msvc-170#reports-and-privacy) the TRX that was produced before? I suspect there is something badly handled related to a fix we did on TRX for another customer. |
|
I don't see the trx fail for my almost empty project when running under WSL. So that is a plus (that it does not fail always). |
|
@nohwnd @Evangelink I can reproduce it locally on my windows machine. You need to specify --results-directory: The error: Unhandled exception. System.IO.IOException: The file 'C:\dev\TestResults\userName_PCName_2024-02-17_11_30_43.trx' already exists. With 1.0.0 it produces correctly all .trx files with milliseconds in the name. With 1.0.1 it produces same milliseconds values so the files have the same name. I have 4 test projects and it is a little bit random => 1+ projects succeed and other get the error. |
Hi,
can anybody share pipeline steps to build/run tests with code coverage and test results?
Or perhaps anybody can help me modify my pipeline to support MSTestRunner?
Thanks!
This is a new project with 3.2.0 runner. What should I change in the pipeline to make it work?
My pipeline:
The text was updated successfully, but these errors were encountered: