New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should generate coverage report files #212

Open
Jaykul opened this Issue Sep 18, 2014 · 40 comments

Comments

Projects
None yet
@Jaykul
Contributor

Jaykul commented Sep 18, 2014

It would be really great if we could export code coverage reports to a reusable format to show progress, and document state.

There are several known formats already (e.g. gcov, lcov, even a json format) -- which are used by projects like codecov.io to generate pretty web-based code-coverage reports, can we pick one of those?

See here, for docs and examples: https://codecov.io/docs

@dlwyatt

This comment has been minimized.

Member

dlwyatt commented Sep 18, 2014

That's a possibility. I'm not sure how to read the lcov files, but gcov is pretty readable. The only problem that I can see is that it seems to give information for an entire line at once... what about lines that contain multiple commands?

return $true; 'This does not execute'; 'This either'

On that line, one statement executed and the other two did not... what should the output file look like?

@Jaykul

This comment has been minimized.

Contributor

Jaykul commented Sep 18, 2014

Yeah, I'm not really happy with them because there aren't any good examples
showing that. In json, true indicates line was paritally hit, but I don't
know about gcov, and I agree with you that this seems insufficient.

http://ltp.sourceforge.net/coverage/lcov/lcov.1.php
http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php

The latter link had information about the lcov format, but it's still very
line-oriented, in all of these there is a basic assumption that putting
multiple commands on one line is bad ;-)

I think the best bet is extending the json format.
https://gist.github.com/stevepeak/d3b49a992415339ea1b0

Joel "Jaykul" Bennett
http://HuddledMasses.org
http://PowerShellGroup.org

On Thu, Sep 18, 2014 at 2:44 PM, Dave Wyatt notifications@github.com
wrote:

That's a possibility. I'm not sure how to read the lcov files, but gcov is
pretty readable. The only problem that I can see is that it seems to give
information for an entire line at once... what about lines that contain
multiple commands?

return $true; 'This does not execute'; 'This either'

On that line, one statement executed and the other two did not... what
should the output file look like?


Reply to this email directly or view it on GitHub
#212 (comment).

@stevepeak

This comment has been minimized.

stevepeak commented Sep 18, 2014

Hey friends! Just dropping in to say we at codecov.io will soon support partial line coverage on select languages that support this concept.

If you have any example files I can check them out to see if they will work properly with our new release. Thanks! We look forward to you joining the family!

@DarqueWarrior

This comment has been minimized.

DarqueWarrior commented Nov 27, 2016

Has there been anymore thought on this? I would really like to publish my code coverage results as part of my CI. What would it take to generate a JaCoCo or Cobertura?

@dlwyatt

This comment has been minimized.

Member

dlwyatt commented Nov 27, 2016

I'll have to look into those formats and see what they support. It's not difficult to take the information we have in memory and dump it out to a file, but Pester's code coverage analysis looks at PowerShell commands, regardless of whether they're one to a line, one command spanning multiple lines, or multiple commands on a single line. Where it gets awkward is translating that to a format that expects a report on what lines were hit.

@stevepeak

This comment has been minimized.

stevepeak commented Nov 27, 2016

@DarqueWarrior Can you upload a Jacoco report to me. I'm curious if if contains complexity data because Codecov will support Jacoco Complexity next month.

Both Jacoco and Cobertura are acceptable upload formats to Codecov.

@DarqueWarrior

This comment has been minimized.

DarqueWarrior commented Nov 27, 2016

@stevepeak Here you go. These are from a very simply Java project created with yo vsts.
CoberturaCodeCoverageReport.zip
JaCoCoCodeCoverageReport.zip

@dlwyatt

This comment has been minimized.

Member

dlwyatt commented Nov 28, 2016

Both of those formats have assumptions from Java (things like "Packages" and "Classes" must exist, etc), but we could probably still just fudge that and put Pester's useful information in there.

@stevepeak

This comment has been minimized.

stevepeak commented Nov 29, 2016

@DarqueWarrior very nice 👍 both those formats work.

@DarqueWarrior

This comment has been minimized.

DarqueWarrior commented Dec 3, 2016

I have used them for ASP.NET Core which does not have Packages but still was useful.

@TravisEz13

This comment has been minimized.

Contributor

TravisEz13 commented Feb 16, 2017

The DSC resource has a Module we use to generate reports for codecov.io here
https://github.com/PowerShell/DscResource.Tests/tree/dev/DscResource.CodeCoverage

@RangerDjanger

This comment has been minimized.

RangerDjanger commented Jun 28, 2017

Has anyone been able to generate code coverage results to publishing into TFS/VSTS?

@it-praktyk

This comment has been minimized.

Contributor

it-praktyk commented Aug 30, 2017

The pull request #782 introduced support for JaCoCo XML code coverage reports - thank you @scriptkiddie123 - so since version 4.0.4-rc Pester supports TFS/VSTS.

Maybe this can be reused for other formats too?

@jonkeda

This comment has been minimized.

jonkeda commented Oct 19, 2017

Just created a version of the JaCoCo xml in pull request #920 and create a JaCoCo reader available at
https://marketplace.visualstudio.com/itemsitemName=Jonkers.JaCoCoreaderfortheJavaCodeCoveragefile.

@scriptkiddie123

This comment has been minimized.

Contributor

scriptkiddie123 commented Oct 20, 2017

@RangerDjanger yes, version 6 of PesterRunner has support for Pester 4, and got a new field to specify where you want the code coverage report: https://marketplace.visualstudio.com/items?itemName=richardfennellBM.BM-VSTS-PesterRunner-Task.

@jonkeda

This comment has been minimized.

jonkeda commented Oct 20, 2017

@scriptkiddie123 Would you be able to test pull request #920 with the PesterRunner? I guess you should be able to see details in VSTS. I haven't been able to try it yet.

@scriptkiddie123

This comment has been minimized.

Contributor

scriptkiddie123 commented Oct 20, 2017

@it-praktyk The code created to do the JaCoCo report is the bare minimum. Mostly because TFS/VSTS only has the 4 bar charts for total files/functions/lines/instuctions, so for my purpose I did not need anything else.

The Get-JaCoCoReportXml function consists of 3 basic parts:
1: Take the code coverage report object, and count the number for hit/missed lines/functions/files, these values are not directly in the report.
2: Defined the JaCoCo xml document.
3: Set the calculated values in the XML.

So to change it to also support Corbetura, you would only need to
1: Move the counting step into the Get-CoverageReport
2: Copy paste the rest of the Get-JaCoCoReportXml to a Get-CobeturaReportXML.
3: Understand the Cobertura format. I can see that it has required fields for branch-rate and complexity, which I don't think Pester provides. Maybe they can be set to null/0, but it would depend on the consumers.
4: Modify the get-coberturaReportXml so it outputs the Cobertura xml.
5: Make a switch in the following code in pester.psm1, that can switch between Cobertura, JaCoCo and other formats:

if (& $script:SafeCommands['Get-Variable'] -Name CodeCoverageOutputFile -ValueOnly -ErrorAction $script:IgnoreErrorPreference) { 
      $jaCoCoReport = Get-JaCoCoReportXml -PesterState $pester -CoverageReport $coverageReport 
      $jaCoCoReport | & $SafeCommands['Out-File'] $CodeCoverageOutputFile -Encoding utf8 
} 
@felixfbecker

This comment has been minimized.

felixfbecker commented Nov 8, 2017

I also found https://github.com/JanJoris/coveralls

It seems like there are a lot of custom scripts. Could one of these get integrated into core so there is some kind of output? It doesn't have to be perfect, it doesn't matter if initially it doesn't report partial hits on lines, but even hit/not hit per line is valuable.

@felixfbecker

This comment has been minimized.

felixfbecker commented Nov 8, 2017

I got it to working in the meantime using the script by DscResource:

As this seems to work fine, could we just integrate that script with a parameter
-CodeCoverageOutputFileFormat json?

@it-praktyk

This comment has been minimized.

Contributor

it-praktyk commented Nov 8, 2017

I'll try to read about coveralls and test it on weekend.
Personally, I'm not sure if we would like to add an external dependency to Pester.

CC: @JanJoris

@felixfbecker

This comment has been minimized.

felixfbecker commented Nov 8, 2017

I'm not saying we should add an external dependency, my understanding is that the JSON format is a commonly used format and I would copy and integrate the DscResources script code and remove any mention of codecov.

@it-praktyk

This comment has been minimized.

Contributor

it-praktyk commented Nov 8, 2017

I think that with that kind of contribution we will not have an issue :-)

Of course

  • the source projects maintainers need to be aware of that. Maybe we should give them even priority to contribute?
  • a source code license needs to be appropriate to a code reuse under Apache license.

@SteveL-MSFT, who can verify formal aspects of reuse the code from the PowerShell\DscResource.Tests repository?

@felixfbecker

This comment has been minimized.

felixfbecker commented Nov 8, 2017

It's MIT. You could also just look at the way they do as an example of how it could be done in Pester.

@jonkeda

This comment has been minimized.

jonkeda commented Nov 9, 2017

In #920 the line details are added for JaCoCo. Looking at the PowerShell\DscResource.Tests it shouldn't be hard to add this.

I don't know how what the proces is for contributing to Pester. Is it ok if I create the code over the weekend? Or is there another way?

@it-praktyk

This comment has been minimized.

Contributor

it-praktyk commented Nov 9, 2017

@jonkeda, please read Contributing to Pester.

I hope to find time to update it soon. An update will contain e.g. what needs to be fulfilled due to PowerShell Core compatibility.

@JanDeDobbeleer

This comment has been minimized.

JanDeDobbeleer commented Nov 15, 2017

Personally, I'm not sure if we would like to add an external dependency to Pester.

Just stumbling in here to see what it's all about. I don't mind if anyone uses my code to add this functionality, I would even be glad to help out as I did a lot of the lifting by trial and error. The only issue today is that, I kid you not, no longer have a proper Windows dev environment due to switching jobs. But I'd gladly create one, because I love Pester :-)

@SteveL-MSFT

This comment has been minimized.

SteveL-MSFT commented Nov 15, 2017

@it-praktyk as noted, it's MIT licensed, so just attribution is required (adding to ThirdPartyNotices.txt is typically what happens)

@nohwnd

This comment has been minimized.

Member

nohwnd commented Nov 15, 2017

@JanJoris You are right on time then. Pester runs cross-platform since this morning. You can either install powershell directly to your linux/mac, or run it in Docker. Either interactively in microsoft/powershell, or use this pre-canned image that I use.

@JanDeDobbeleer

This comment has been minimized.

JanDeDobbeleer commented Nov 20, 2017

@nohwnd cool, let me try that!

@felixfbecker

This comment has been minimized.

felixfbecker commented Nov 20, 2017

I did some slight modifications to the DSC CodeCov script to make it work cross-platform for use in ps-nvm: https://github.com/aaronpowell/ps-nvm/blob/bf4ca1953b250c11887a36464de566794947afb1/.scripts/CodeCovIo.psm1
The original is linked at the top if you want to diff it.
It's just stuff like git.exe -> git and handling forward slashes in file paths

@nohwnd

This comment has been minimized.

Member

nohwnd commented Dec 17, 2017

@JanJoris Still willing to contribute this? :)

@stevepeak

This comment has been minimized.

stevepeak commented Dec 18, 2017

(Codecov Team Member) I'll be happy to help verify uploads and json structure when it's time. Just ping me. 👍

@VertigoRay

This comment has been minimized.

VertigoRay commented Apr 11, 2018

@felixfbecker Thanks for sharing your process. I set it up, but it gave me 0% for the Composite Resource I'm working on. Mind looking at the repo and letting me know if anything stands out to you?

https://github.com/VertigoRay/OneDriveDsc

@felixfbecker

This comment has been minimized.

@VertigoRay

This comment has been minimized.

VertigoRay commented Apr 11, 2018

@felixfbecker You said it was minor changes for linux/mac compatibility, so I didn't switch to that. I took that one from your first post:
https://github.com/aaronpowell/ps-nvm/blob/9784e5ada938a7485a6786371ac76b3ed78d37fb/appveyor.yml#L15

I'll switch to the one you have posted just above and see if it works.

@VertigoRay

This comment has been minimized.

VertigoRay commented Apr 11, 2018

@felixfbecker Same issue with newer script. I don't want to clutter this issue, but I want to be sure that whatever solution is implemented properly tracks coverage of DSC Composite Resources.

https://codecov.io/gh/UNT-CAS/OneDriveDsc

@it-praktyk

This comment has been minimized.

Contributor

it-praktyk commented May 14, 2018

This issue was referenced in the blog post Adding a code coverage badge to a PowerShell project by @markwragg

@it-praktyk

This comment has been minimized.

Contributor

it-praktyk commented Jul 24, 2018

I updated the pull request #920. Now the proposed feature (detailed code coverage) is updated to be compatible with PowerShell Core 6.x too.

Before merge tests are needed. Can you help?

@felixfbecker

This comment has been minimized.

felixfbecker commented Aug 12, 2018

I published the script I mentioned above in a module: https://www.powershellgallery.com/packages/PSCodeCovIo

It exports the $res.CodeCoverage data to a JSON file that can be uploaded to CodeCov with the standard bash uploader.

@Mobrockers

This comment has been minimized.

Mobrockers commented Oct 22, 2018

We would like to include our pester code coverage results in our sonarqube projects. SonarQube uses their generic test data format for uploading test data. The format is XML and they provide an xsd to validate against. Information can be found here: https://docs.sonarqube.org/display/SONAR/Generic+Test+Data
We would very much like this format to be available as output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment