Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support non-binary format for Code Coverage #2874

Closed
jakubch1 opened this issue Apr 26, 2021 · 40 comments
Closed

Support non-binary format for Code Coverage #2874

jakubch1 opened this issue Apr 26, 2021 · 40 comments

Comments

@jakubch1
Copy link
Member

Description

Support non-binary format for dynamic code coverage. Code coverage is supported on ubuntu and alpine but users can't open coverage report on this platforms as .coverage files can be opened only in VS.

Created based on feedback from users on #981

@fhnaseer
Copy link
Member

fhnaseer commented Nov 19, 2021

Cobertura support is added to code coverage. It is available in the following version. https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.1.0-preview-20211118-03

To get cobertura report, add <Format>Cobertura</Format> in the runsettings.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
           <Format>Cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Available format options:

  1. Coverage (default binary file format)
  2. Cobertura
  3. Xml (Same as codecoverage.exe /analyze report.coverage output).

@ElvenSpellmaker
Copy link

Hey @jakubch1 and @fhnaseer, any idea when this will land in stable rather than preview please (for Linux Cobertura/coverlet support)?

@jakubch1
Copy link
Member Author

Stable 17.1.0 packages will be released in February.

@ElvenSpellmaker
Copy link

@jakubch1 Thanks!

@ManikandanMani
Copy link

ManikandanMani commented Mar 8, 2022

Cobertura support is added to code coverage. It is available in the following version. https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.1.0-preview-20211118-03

To get cobertura report, add <Format>Cobertura</Format> in the runsettings.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
           <Format>Cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Available format options:

  1. Coverage (default binary file format)
  2. Cobertura
  3. Xml (Same as codecoverage.exe /analyze report.coverage output).

Hi @fhnaseer & @jakubch1, I currently use version 17.1.0 of microsoft.codecoverage package which targets net6.0 Test project and I have updated the format as xml and cobertura in runsettings and it generated both formats xml file, but looking those file did gave only very minimal data and I am not sure how to use this data for coverage of whats actually covered, could you let us know, is this expected coverage report or do I need to use preview versions of the packages?

image
TestProjectPackages

@jakubch1
Copy link
Member Author

@ManikandanMani on what OS you collect code coverage?

@ManikandanMani
Copy link

@jakubch1 I am trying to generate this on windows 10.

@jakubch1
Copy link
Member Author

@ManikandanMani could you please add to you command line: --diag logs.txt and provide all files logs*.txt to us.

@ManikandanMani
Copy link

ManikandanMani commented Mar 15, 2022

@jakubch1, I have attached all the log files that were generated using this command to run test in powershell, But if we specify two formats like <Format>Xml;Cobertura</Format> in run settings file, it generates only the visual studio coverage file, not the specified coverage xml files aren't generated.

  1. & "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" --collect:"Code Coverage" D:\TestProject\bin\Debug\net6.0\TestProject.dll /TestCaseFilter:"Name=TestName" /Settings:"D:\TestProject\vstestcobertura.runsettings" /Diag:"D:\TestProject\vstestcobertura01.txt"

  2. Test Run Settings

     <DataCollectors>
       <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
         <Configuration>
           <Format>Cobertura</Format>
           <!-- <Format>Xml;Cobertura</Format> -->
           <CoverageFileName>TestProject</CoverageFileName>
             <CodeCoverage>
               <ModulePaths>
                 <Exclude>
                   <ModulePath>.*NUnit.Framework.dll</ModulePath>
                   <ModulePath>.*NUnit3.TestAdapter.dll</ModulePath>
                   <ModulePath>.*TestAdapter.*</ModulePath>
                 </Exclude>
             </ModulePaths> 
           </CodeCoverage>
         </Configuration>
       </DataCollector>
     </DataCollectors>
 </DataCollectionRunSettings>

vstestcobertura01.host.22-03-15_21-55-54_07155_9.txt
vstestcobertura01.txt
vstestcobertura01.datacollector.22-03-15_21-55-51_93742_5.txt

@jakubch1
Copy link
Member Author

Please verify on your computer that below files exists:

C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Platform\InstrumentationEngine\x64\MicrosoftInstrumentationEngine_x64.dll
C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Platform\InstrumentationEngine\x86\MicrosoftInstrumentationEngine_x86.dll
C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\..\..\Team Tools\Dynamic Code Coverage Tools\VanguardInstrumentationProfiler_x86.config
C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\..\..\Team Tools\Dynamic Code Coverage Tools\amd64\VanguardInstrumentationProfiler_x64.config

@ManikandanMani
Copy link

ManikandanMani commented Mar 16, 2022

Of these 4 file paths, first 2 exists, rest 2 doesn't exists, could you share the absolute file path for the ones below, do I need to locate it under here C:\Program Files\Microsoft Visual Studio\2022\Community\Team Tools or and are these only for vs2022 enterprise not for community 2022 ?

C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\..\..\Team Tools\Dynamic Code Coverage Tools\VanguardInstrumentationProfiler_x86.config
C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\..\..\Team Tools\Dynamic Code Coverage Tools\amd64\VanguardInstrumentationProfiler_x64.config

image

@jakubch1
Copy link
Member Author

I think those files are only part of Enterprise edition. Please use this package to run your tests:
https://www.nuget.org/packages/Microsoft.TestPlatform/17.1.0
You should have all files there.

@fhnaseer
Copy link
Member

@jakubch1, I have attached all the log files that were generated using this command to run test in powershell, But if we specify two formats like <Format>Xml;Cobertura</Format> in run settings file, it generates only the visual studio coverage file, not the specified coverage xml files aren't generated.

  1. & "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" --collect:"Code Coverage" D:\TestProject\bin\Debug\net6.0\TestProject.dll /TestCaseFilter:"Name=TestName" /Settings:"D:\TestProject\vstestcobertura.runsettings" /Diag:"D:\TestProject\vstestcobertura01.txt"
  2. Test Run Settings
     <DataCollectors>
       <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
         <Configuration>
           <Format>Cobertura</Format>
           <!-- <Format>Xml;Cobertura</Format> -->
           <CoverageFileName>TestProject</CoverageFileName>
             <CodeCoverage>
               <ModulePaths>
                 <Exclude>
                   <ModulePath>.*NUnit.Framework.dll</ModulePath>
                   <ModulePath>.*NUnit3.TestAdapter.dll</ModulePath>
                   <ModulePath>.*TestAdapter.*</ModulePath>
                 </Exclude>
             </ModulePaths> 
           </CodeCoverage>
         </Configuration>
       </DataCollector>
     </DataCollectors>
 </DataCollectionRunSettings>

vstestcobertura01.host.22-03-15_21-55-54_07155_9.txt vstestcobertura01.txt vstestcobertura01.datacollector.22-03-15_21-55-51_93742_5.txt

You can specify only one output format.

@ManikandanMani
Copy link

I added this package https://www.nuget.org/packages/Microsoft.TestPlatform/17.1.0, and ran the tests and it generated the coverage xml file same as old, and I also tried by adding the test adapter path of coverlet.collector /TestAdapterPath:"$Env:userprofile\.nuget\packages\coverlet.collector\3.1.0\build\netstandard1.0\", but it too generated the same like coverage xml file.
image

xml files generated after adding package https://www.nuget.org/packages/Microsoft.TestPlatform/17.1.0
image

@fhnaseer
Copy link
Member

Can you please share logs? Also if you generate .coverage file, is it also empty?

@ManikandanMani
Copy link

@fhnaseer, I have shared the logs above for collect coverage with cobertura and vstest, yes the generated .coverage file seems empty when I used VsTest.Console.exe from VS2022 program files, since I can't view the .coverage file with vs2022 community I converted with CodeCoverage.exe and it was empty. I have attached the diag logs generated for visual studio coverage, codecoveragexml file.

vstestcobertura01.host.22-03-17_10-43-31_62605_9.txt
vstestcobertura01.datacollector.22-03-17_10-43-28_97451_5.txt
vstestcobertura01.txt

Generated .CoverageXml file from .Coverage, I couldn't able to attach the coveragexml , xml files here.
image

@fhnaseer
Copy link
Member

It seems like you are using vstest.console.exe from Visual Studio (Community Edition). In the datacollector.txt
TpTrace Verbose: 0 : 10952, 1, 2022/03/17, 10:43:30.884, 25358358353, datacollector.exe, TestPluginCache: Using directories for assembly resolution 'C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\Extensions,C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\Extensions\TestPlatform\Extensions\V1'.

Please use the one from nuget package.
D:\Downloads\microsoft.testplatform.17.1.0\tools\net451\Common7\IDE\Extensions\TestPlatform\vstest.console.exe /Diag:logs.txt /Collect:"Code Coverage" .\TestProject17.dll

Also you can try dotnet test --collect "Code Coverage"

@ManikandanMani
Copy link

@fhnaseer, I thought this would work directly when using with vstest.console.exe from Visual Studio (Community Edition) and by referencing microsoft.codecoverage nuget package in the test project, do you guys able to generate cobertura xml file directly by pointing to vstest.console.exe comes with Visual Studio 2022[community/enterprise]? or the only way to generate cobertura xml file is to target to vstest.console.exe from microsoft.testplatform.17.1.0 nuget package ? kindly let me know am I missing anything here?

@fhnaseer
Copy link
Member

If you have Visual Studio Enterprise then you can generate code coverage either using UI commands inside VS or by running vstest.console.exe.
In case of Community edition, it will not work as code coverage related package is shipped only in enterprise edition.

So if you have community edition, there are two ways to generate coverage report

  1. vstest.console.exe from microsoft.testplatform.17.1.0 package'
  2. dotnet test --collect "Code Coverage" command

@ManikandanMani
Copy link

If I have Visual Studio Enterprise Can I able to generate cobertura coverage xml directly vstest.console.exe, without specifying the vstest from microsoft.testplatform.17.1.0 package?

@fhnaseer
Copy link
Member

fhnaseer commented Mar 18, 2022

Yes. You need to specify runsettings file when Analyzing Code Coverage.

@ManikandanMani
Copy link

Thanks @fhnaseer, I could able to generate cobertura xml as coverage file when I used enterprise Vstest.console.exe

@ManikandanMani
Copy link

@fhnaseer, could you let us know how the coverage file will be generated for test reruns if its failed at the first time and passed at the next run?

@fhnaseer
Copy link
Member

Foreach test run a separate coverage file will be generated. Coverage report is not dependent on test results. So test fail or pass coverage file will be generated. Coverage reports will be generated in "Test Results" folder in your solution.

@ManikandanMani
Copy link

And does this feature supports only Visual studio version 17.0/17.1 ?

@fhnaseer
Copy link
Member

Support of cobertura was added in 17.1.0.

@ManikandanMani
Copy link

ManikandanMani commented Mar 18, 2022

@fhnaseer, could you able to publish this xml file in the azure devops build pipelines after the test ran using the vstest task version-2, In the vstest task after the tests are completed before starting the next build step the vstest tasks deletes the TestResults folder, and the ResultsDirectory tag in the runsettings file is also overridden to the $(Agent.TempDirectory)\TestResults in the build, and does generating coverage xml works on vs2019 enterprise latest version?
image

@fhnaseer
Copy link
Member

Support for cobertura and xml file was not supported in 17.0 or earlier versions. For azure devops and code coverage you can follow the following article. https://docs.microsoft.com/en-us/azure/devops/pipelines/test/review-code-coverage-results?view=azure-devops

@ManikandanMani
Copy link

Hi @fhnaseer, When the code coverage file format is either cobertura xml or coverage xml, In vstest version-2 task when code coverage checkbox is enabled or disabled, it wipes the test results directory after the test execution completed but it doesn't shows the coverage xml as the build artifact like the .coverage file generated as a artifact.
I had to get this file using 2 api endpoints and download this, then use it under publish code coverage task.
Agent version: 2.192.0

1.https://docs.microsoft.com/en-us/rest/api/azure/devops/testresults/testlogstoreendpoint/get-test-log-store-endpoint-details-for-run-log?view=azure-devops-rest-6.0
2.https://docs.microsoft.com/en-us/rest/api/azure/devops/test/runs/query?view=azure-devops-rest-6.0

@MarcoRossignoli
Copy link
Contributor

@ManikandanMani you can combine VSTest@2 and PublishCodeCoverageResults@1 to publish code coverage like:


  - task: VSTest@2
    inputs:
      displayName: "VsTest - testAssemblies"
      inputs:
      testAssemblyVer2: |
        **\bin\Debug\net6.0\TestProject*.dll
        !*TestAdapter.dll
        !**\obj**
      resultsFolder: '$(build.ArtifactStagingDirectory)/TestResults'
      otherConsoleOptions: /collect:"Code Coverage;Format=Cobertura"

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: $(build.ArtifactStagingDirectory)/TestResults/*/*.xml

If you want it inside the artifacts to download you can combine VSTest@2 to generate the coverage file, CopyFiles@2 to move to $(Build.ArtifactStagingDirectory) and PublishPipelineArtifact@1 to publish to the pipelines artifacts.

@ElvenSpellmaker
Copy link

Hey @jakubch1, is there a non-preview version of this available yet?

@jakubch1
Copy link
Member Author

Yes:

jakubch@Jachocho-5820:~/examples/t1$ dotnet --version
6.0.300
jakubch@Jachocho-5820:~/examples/t1$ cat t1.csproj |grep Test.Sdk
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
jakubch@Jachocho-5820:~/examples/t1$ dotnet test --collect "Code Coverage;Format=cobertura"
  Determining projects to restore...
  All projects are up-to-date for restore.
  t1 -> /home/jakubch/examples/t1/bin/Debug/net6.0/t1.dll
Test run for /home/jakubch/examples/t1/bin/Debug/net6.0/t1.dll (.NETCoreApp,Version=v6.0)
Microsoft (R) Test Execution Command Line Tool Version 17.2.0 (x64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Attachments:
  /home/jakubch/examples/t1/TestResults/0f72fb94-6e69-4641-8fcc-22962a49d06e/jakubch_Jachocho-5820_2022-05-23.10_17_29.cobertura.xml
Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: < 1 ms - /home/jakubch/examples/t1/bin/Debug/net6.0/t1.dll (net6.0)

@ManikandanMani
Copy link

@ManikandanMani you can combine VSTest@2 and PublishCodeCoverageResults@1 to publish code coverage like:


  - task: VSTest@2
    inputs:
      displayName: "VsTest - testAssemblies"
      inputs:
      testAssemblyVer2: |
        **\bin\Debug\net6.0\TestProject*.dll
        !*TestAdapter.dll
        !**\obj**
      resultsFolder: '$(build.ArtifactStagingDirectory)/TestResults'
      otherConsoleOptions: /collect:"Code Coverage;Format=Cobertura"

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'Cobertura'
      summaryFileLocation: $(build.ArtifactStagingDirectory)/TestResults/*/*.xml

If you want it inside the artifacts to download you can combine VSTest@2 to generate the coverage file, CopyFiles@2 to move to $(Build.ArtifactStagingDirectory) and PublishPipelineArtifact@1 to publish to the pipelines artifacts.

@MarcoRossignoli , the vstest task will delete the test results directory specified in the task options, so the only way I found is through the api's.

@MarcoRossignoli
Copy link
Contributor

MarcoRossignoli commented Jun 20, 2022

the vstest task will delete the test results directory specified in the task options, so the only way I found is through the api's.

I think that if the result folder points to a staging directory will be preserved between tasks https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables-devops-services
I don't expect any "cleanup" by VSTest task, resultsFolder should be simply passed to the underneath vstest.console.exe as parameter.

@ManikandanMani
Copy link

I use the classic version,If you ran the pipeline in debug mode you could able to see the logs to deleting test results attachments, I hope I haven't missed anything.
##[debug]Object of TestRunPublisher created. ##[debug]Entering CreateTestRunAttachmentsAsync ##[debug]Leaving CreateTestRunAttachmentsAsync ##[debug]Uploading test run attachements individually ##[debug]Entering UploadTestRunLogAsync ##[debug]Blob name: GeneralAttachment/projects.cobertura.xml . ##[debug]Entering UploadTestRunLogAsync ##[debug]Blob name: GeneralAttachment/XYZ02_2022-06-20_07_44_37.trx . ##[debug]File uploaded successsfully on LogStore XYZ02_2022-06-20_07_44_37.trx ##[debug]File uploaded successsfully on LogStore projects.cobertura.xml ##[debug]Published test run level attachments ##[debug]PERF: Publish:Attachments: took 932.0522 ms ##[debug]Publish.Run : Publishing to tcm finished ##[debug]PERF WARNING: Publish:Run: took 3326.199 ms ##[debug]ExecuteAndPublish.Run : Cleanup invoked. ##[debug]Cleanup.Cleanup : Creating new cleanup instance. ##[debug]Cleanup.Run : Initialize cleanup ##[debug]Processed: ##vso[task.uploadfile]C:\Agent\*.22-06-20_07-42-18_97119_7.diaglog ##[debug]Processed: ##vso[task.uploadfile]C:\Agent\*\Slice-Execution-6122_1dlqow.diaglog ##[debug]Processed: ##vso[task.uploadfile]C:\Agent\*\Slice-Execution-6122_1dlqow.host.22-06-20_07-42-14_68405_9.diaglog ##[debug]Processed: ##vso[task.uploadfile]C:\Agent\*\Slice-Execution-6122_1dlqow.host.22-06-20_07-42-21_07634_10.diaglog ##[debug]Processed: ##vso[task.uploadfile]C:\Agent\*\TraceLogs-Execution-6122_1dlqow.diag ##[debug]Cleanup.Run : Deleting results directory : C:\Agent\*\_work\169\a\TestResults ##[debug]Cleanup.Run : Cleanup finished ##[debug]PERF WARNING: ExecutionAndPublish:Run: took 159376.9748 ms ##[debug]ExecutionStateResolver.SetNextPhase : Current Phase: Execution ##[debug]ExecutionStateResolver.SetNextPhase : Processing Execution Restults : Updating the slice with execution results. SliceId: 6122. ##[debug]ExecutionStateResolver.SetNextPhase : Next Phase: SliceFetch ##[debug]TestExecutionHost.Execute: Current Phase SliceFetch ##[debug]VstestAdapter.ExecuteAsync : Starting Execution ##[debug]VstestAdapter.ExecuteAsync : Returning ##[debug]TestExecutionServiceListener.Start invoked

@MarcoRossignoli
Copy link
Contributor

I use the classic version

What you mean for classic version?

@ManikandanMani
Copy link

I use the classic version

What you mean for classic version?

Azure devops build pipeline template type

@ManikandanMani
Copy link

@MarcoRossignoli, can we able to differentiate coverage files been generated for test reran, at present we couldn't able to.

Result Attachments will be stored in LogStore
Publishing test results to test run '1008516'.
TestResults To Publish 3, Test run id:1008516
Test results publishing 3, remaining: 0. Test run id: 1008516
Published test case results: 3
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
##[warning]This cobertura.xml already exists
Failed to upload run logs to LogStore, Trying to upload to FileService
Received the command : Stop
TestExecutionHost.ProcessCommand. Stop Command handled
SliceFetch Aborted. Moving to the TestHostEnd phase

@MarcoRossignoli
Copy link
Contributor

@ManikandanMani can you show your yaml tasks it's not clear to me what you mean or if you've private information you can open a ticket here https://developercommunity.visualstudio.com/

@ManikandanMani
Copy link

ManikandanMani commented Jul 5, 2022

@ManikandanMani can you show your yaml tasks it's not clear to me what you mean or if you've private information you can open a ticket here https://developercommunity.visualstudio.com/

steps:

  • task: VSTest@2
    displayName: 'Smoke Tests web - CI'
    inputs:
    testAssemblyVer2: |
    **\$(BuildConfiguration)\*\*$(ProjectFileName).dll
    !*TestAdapter.dll
    !
    \obj**
    testFiltercriteria: 'TestCategory=SmokeSuite'
    runSettingsFile: 'Test/vstestcobertura.runsettings'
    runInParallel: true
    codeCoverageEnabled: true
    distributionBatchType: basedOnAssembly
    testRunTitle: 'Smoke Tests web - CI'
    rerunFailedTests: true
    rerunMaxAttempts: 1

stan-sz added a commit to stan-sz/csharp that referenced this issue Jan 16, 2023
Thanks to microsoft/vstest#2874 Cobertura reports can be obtained faster without coverlet.
k8s-ci-robot pushed a commit to kubernetes-client/csharp that referenced this issue Feb 11, 2023
* Use vstest Cobertura support

Thanks to microsoft/vstest#2874 Cobertura reports can be obtained faster without coverlet.

* Update Kubectl.Tests.csproj

* Update KubernetesClient.Classic.Tests.csproj

* Update KubernetesClient.Tests.csproj

* Update buildtest.yaml

* Update AbstractKubernetes.cs.template

* Update IBasicKubernetes.cs.template

* Update IOperations.cs.template

* Update Model.cs.template

* Add filter

* revert
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants