Generate Code Coverage of Piwik test suite

[mattab]

It would be fantastic to know how much of the piwik platform source code is covered by tests. With our hundreds of unit tests, [integration tests, and http://piwik.org/blog/2013/10/our-latest-improvement-to-qa-screenshot-testing/ UI screenshot tests, we can hope for a reasonnable code coverage.

Access to code coverage report would help us find out if there are any parts of the code that are not tested at all, and it will help quantify the reach of our QA suite.

[mattab]

• it would be mostly useful only if it included the coverage from Core+Plugins+Integration
• and +1 if we had also the UI tests coverage since they hit UI + controllers!

But not sure how/if it is possible to "merge" the code coverage from several phpunit executions?

[mattab]

In 7567afa: Removing code coverage refs #4224

[sgiehl]

I've started to work on that ticket. There is a possibility to "merge" the code coverage.
With phpcov (https://github.com/sebastianbergmann/phpcov) it is possible to merge coverage reports generated in serialized php format (--coverage-php option).
It seems to work so far but it will take quite much ram to combine those reports. 256 MB weren't enough to combine core and plugin tests.

[sgiehl]

In cab7c14: refs #4224 include uncovered files to code coverage

[mattab]

See follow up ticket to this one: #4302 Setup Coveralls for Piwik Code coverage report

@SteveG hopefully your changes to make Code coverage work could be used on coveralls.

[sgiehl]

As far as I've seen coveralls requires some CI tool running, such as jenkins or travis.
It shouldn't be a problem to run that on an jenkins, but we would need to reinstall that again. I'm not sure if we would be able to do that with travis. Currently there are 3GB ram required to merge the code coverage reports - and all tests + code coverage merging is running around 1,5 - 2 hours.

[mattab]

Wow very cool to hear you managed to build it with 3g of ram.

code coverage will not run well on travis, and we cant run it on piwik.org as it would slow down services, maybe we could run this process in a large EC2 instance and shut it down after... would be cheap!

[sgiehl]

In e9f5428: refs #4224 do not check auto prepend/append config in test mode, as it is used to generate full code coverage

[mattab]

@Steve, would it be possible to upgrade the codecoverage report from Nov 22 on 2.0.3 ?

Here is an idea: maybe we could run the whole processus, once a day, generate the coverage at midnight. Maybe we can use the existing travis script to setup webserver+mysql etc. and upload results to codecoverage.piwik.org.

Maybe we could run the script on AWS once a day?

[sgiehl]

I'm currently having problems with running the UI tests. Setting up the fixtures always fail with Failed asserting that 'http://local/piwik/tests/PHPUnit/proxy/piwik.php?idsite=1&rec=1&apiv=1&r=795241&cip=156.5.3.2&_id=76c0cff81faceb99&cdt=2012-01-12+00%3A11%3A42&token_auth=c0e024d9200b5705bc4804722636378a&_idts=1389624137&_idvc=0&fla=1&java=1&dir=0&qt=0&realp=0&pdf=0&wma=0&gears=0&ag=0&h=12&m=34&s=6&res=1024x768&cookie=1&url=http%3A%2F%2Fexample.org%2Findex.htm%3Futm_campaign%3DGA+Campaign%26piwik_kwd%3DPiwik+kwd%26utm_term%3DGA+keyword+SHOULD+NOT+DISPLAY%23pk_campaign%3DNOT+TRACKED%21%21%26pk_kwd%3DNOT+TRACKED%21%21&urlref=' contains "_viewts=1302307497".

Nevertheless - I'll try to commit the required files and write a small description how to setup/run the complete code coverage by the end of the week.

[mattab]

@SteveG we have fixed this test error last week, maybe now it works fine for you too?

[mattab]

@sgiehl can this be closed now that coverage is enabled? 👍

[sgiehl]

Currently only directly called methods are covered by the report. So all methods called through the tracker/api proxy will not be included. It's not easy to do that. But there is a way, which costs much performance (and will exceed the travis resources). If we don't need that anymore. we can close this ticket.

[mattab]

Kuddos @sgiehl for making it happen.

hopefully more and more devs will in the future find useful improvements to testing thanks to these code coverage reports.