"Stable" format for coverage

[tmysik]

Currently, code coverage generates an HTML page which is good for humans. But for other tools (like NetBeans in my case ;), it would be much better to have e.g. XML file, preferably in Clover format (this format is widely used and contains information that perfectly suits our needs).

Thanks.

[dg]

The only documentation I found for clover format https://gist.github.com/aik099/8556655

[tmysik]

I have never found any XSD/DTD for it - shame on me (or Atlassian? ;), I know but it was the only usable format for coverage (at least for us) for PHP (PHPUnit) and JavaScript (karma-coverage, or better Istanbul). It has more relevant information than e.g. Cobertura.

Sample for PHPUnit can be found here [1], generator for Istanbul here [2]. I hope it can help.

Thanks.
[1] http://hg.netbeans.org/web-main/file/f749e209fe25/php.phpunit/test/unit/data/phpunit-coverage.xml
[2] https://github.com/gotwarlost/istanbul/blob/f2c84f99739006ec657be45f45ac35f8ad377e13/lib/report/clover.js

[tmysik]

One note - the PHPUnit report would be perfectly fine for us.

Thanks.

[tmysik]

BTW if you are able to implement it quite soon, I could add support for it for the next NetBeans version (its feature freeze is around July 5th I think).

Thanks.

[milo]

To be sure, you are talking about Clover format developed by the Atlasian company for Java.

I found topic about Clover XSD schema and I found download of Clover-for-Ant which contains mentioned XSD.

I didn't study the schema in detail, but at first glance it requires more information than Tester is possible easily provide now.

Next question about this format is licensing and format stability. The schema I found has not own namespace nor versioning.

@tmysik Don't you know about an another (open) standard? :)

[tmysik]

@milo As I wrote, this format suits best to our needs; for each file, we need:

• file path
• line count
• number of statements
• number of covered statements
• hit count for covered lines

See [1] for more information.

So, to your questions:

To be sure, you are talking about Clover format developed by the Atlasian company for Java.

Yes.

I didn't study the schema in detail, but at first glance it requires more information than Tester is possible easily provide now.

Yes, it is complex. That is why I wrote that the same "minimal" report as PHPUnit generates now (see my prevous comment for an example) is perfectly fine for us. I know that PHPUnit uses Xdebug extension for it.

Next question about this format is licensing

Not a question for me, sorry, I have no idea. But I would suppose that it is fine to generate this format (since PHPUnit and also other tools do it).

and format stability. The schema I found has not own namespace nor versioning.

Again, I am not able to answer this, sorry.

Don't you know about an another (open) standard? :)

As I wrote, e.g. Istanbul supports these formats [2]:

• HTML
• LCOV
• Cobertura
• TeamCity
• Clover

For us, only HTML and Clover are usable (contain information we need). HTML is bad because it is very easy to change it so any parser can be easily broken.

Just to be sure - we do not need Clover, we need any (likely XML) format which provides information I have described above. We are totally OK with your own format if you are going to create any.

Thanks guys.
[1] http://hg.netbeans.org/web-main/file/tip/web.clientproject.api/src/org/netbeans/modules/web/clientproject/api/jstesting/Coverage.java
[2] https://github.com/gotwarlost/istanbul

[TomasVotruba]

clover format is also only one accepted by scrutinizer (command detail), since it's default output of phpunit.

So at the moment there's no way to get that nice "test coverage badge".

Currently supported formats are only html and serialized string, aren't they?

[milo]

I was discussing that on PoSobota and the Clover format is really one of the most spreaded out. This is WIP now.

[TomasVotruba]

Great! This might help: https://github.com/sebastianbergmann/php-code-coverage/blob/master/src/CodeCoverage/Report/Clover.php

[TomasVotruba]

Just FYI, I've decided to use PHPUnit due to simple clover generation, so this issue is solved for me.

[matej21]

I've tried this as a replacement of html template and it works great with the phpstorm...

[TomasVotruba]

@matej21 Do you have some repo where you use it with scrutinizer?

Why not PR?

[milo]

I sank into the issue. The problem is, I didn't find any official Clover schema/specification. The only available XSD I found is from Clover for Ant project (it is the same as in the David's first comment). The question is, how much the XSD describes real requirements, because we are not able to achive them all (not nearly). Because an XML coverage report must:

• use some name (easy)
• describe project sources and tests sources separately (not so hard)
• expose metrics like count of conditionals, elements, statements, methods and their covered equivalents
• these metrics are mandatory for every project (only one in our case), package (let's say namespace), file and class

These all are required by mentioned XSD, I didn't write about optionals. PHPUnit exports some of these but it is not XSD compatible too.

So, implementation is about compromise for now. And the easiest is the @matej21's one.

So, @tmysik, @TomasVotruba, @Vrtak-CZ... do you have some relevant links for me, where is a some Clover compatibility description? For tools Netbeans, PhpStorm, Sruntinizer, Jenkins CI.

[JanTvrdik]

I would suggest using the simple approach as suggested by @matej21, release it as experimental feature and see how it will work in real world.

[TomasVotruba]

Either @JanTvrdik 👍 or mimic PHPUnit output I linked.

[Vrtak-CZ]

Sorry I didn't have any info about schema / specification. But I agree with @JanTvrdik

[tmysik]

@milo I already answered what would be the best from the NetBeans' point of view, see my comment from July 24th (#151 (comment)).

[milo]

@matej21 How does it work with phpStorm for you? I mean, how do you load the generated XML by phpStorm?

[matej21]

@milo Tools/Show code coverage data

[milo]

Indeed! Thanks.

[milo]

WIP PR #196

[jkuchar]

This looks like ready for closing :)