Test coverage badge / report

[steffenmllr]

First off thanks for done, it's a great project. I'm testing a couple of nodejs projects at the moment and was asking myself if there is a way to display my coverage results ?

I'm thinking something in the line of:

coverage:
  - npm run coverage

Which produces a stdout in percent and gets saved to the sqlite and displayed in a nice bagde. Is there a way to make that happen ?

[bradrydzewski]

Yes, this is a feature that I'd love to add.

We'll need to parse the coverage report file(s) as opposed to parsing stdout. Luckily Docker exposes an endpoint to extract files: http://docs.docker.io/en/latest/reference/api/docker_remote_api_v1.10/#copy-files-or-folders-from-a-container

The challenge is there are multiple formats for coverage reports. There are some widely used formats, such as:

• cobertura (java, npm)
• lcov (npm)
• clover (java, phpunit)

Some langauges even have their own custom formats. Go, for example, has coverage reporting built-in to the standard library and uses a custom plain-text format.

Therefore I think the .drone.yml syntax could like like this:

report:
  # syntax for extracting clover coverage
  clover:
     - path/to/file1.txt
     - path/to/file2.txt

  # syntax for extracting cobertura coverage
  cobertura:
     - path/to/file1.xml
     - path/to/file2.xml

  ...

We would have a "plugin" for each format that is capable of extracting the overall coverage percentage. Eventually I'd like to extract the package-level and file-level coverage details as well.

[marcosnils]

Cool feature.

Just one question, does it make sense to configure multiple coverage tools per .drone.yml file?. If so, drone would have the responsibility to gather all the results and then merge them to get one single result.

What about if drone only supports one coverage tool in the beginning and the .drone.yml is something like this?.

report:
  # regex syntax to extract values
  source:
    - path/to/file.txt
    - path/to/folder/ # 

[bradrydzewski]

The report section of the .drone.yml should be flexible enough to support more than just coverage reports, in case we ever want to support other types of metrics. For example:

report:
    junit:
       - path/to/junit.xml
    cobertura:
       - path/to/cobertura.xml

Supporting multiple coverage formats shouldn't be too difficult. We will just need to persist the results in the Drone database in a common format. This might sound difficult, but the Coveralls team has already figured this out, and we can use their format as a baseline.

Aggregating results would be possible, but also might be overkill. Let's say (for whatever reason) someone generates two different coverage reports in two different formats. We could display two different coverage reports in the UI and offer two different badges:

# badge for cobertura
http://drone.io/badges/github.com/drone/drone/cobertura.svg
# badge for gocov
http://drone.io/badges/github.com/drone/drone/gocov.svg

I think we can start with one format (let's say Cobertura) however, we can make it easy to add other formats in the future.

[alexandre-butynski]

+1 for reports about code coverage (we are using clover with phpunit) and, more generally, to allow development of extensions for all kind of analysis.

For example, in the PHP world, we have a lot of small tools for static analysis (phploc, phpmd, phpcpd...) that produce great xml and/or html reports. It could be a very small step for printing pretty results in Drone and/or redirect to html reports hosted somewhere... Is that a dream ?

[steffenmllr]

I would love to help but I'm relatively new to go. So if someone would start with a sample report plugin I'll try to adapt it to some nodejs coverage tools (istambul, lcov, lab)

[alexandre-butynski]

I'm like @steffenmllr. I would be happy to work on php report plugin if a generic interface was provided.

[bradrydzewski]

I really appreciate everyone's offer to help. Unfortunately we're just not quite ready to implement this feature yet. There is a lot of non-coding work that needs to go into this, and we have some higher priority items we need to tackle first.

This is definitely one of the features I'm most excited about (I love Go's coverage tool) so it will definitely happen, it just might take a month or two.

[marinho]

(yep, I am new here, but I have followed the discussion for a while and this a topic that interests me a lot).

Well, even knowing it is not a priority now (the message is clear), what if Drone doesn't care about formats and just calls a line command that outputs a number?

So, no matters which language, framework or format, as long as the output is a number, Drone catches it. So, for each language we can have plugins or customized scripts, not a Drone's business.

[bradrydzewski]

Go, for example, generates a coverage % for each package. So we can't rely on just a single number. I think parsing the command line, and creating plugins to parse the command line, would end up being more complex than extracting and parsing structured files.

[marinho]

I see, but what I mean is: Drone expects for a number, and a script for each language/format will be responsible to do whatever and output the number. In case of Go, this script will convert the % into the number. That's what I mean

[marinho]

My point is that, there are so many languages, formats, frameworks, etc. that it wouldn't be sensitive for Drone to do it, and even very difficult. If you transfer this responsibility to an external script, you allow easy customization and implementation.

In my case for example, only Python and Go matter, so, I would care about implementations for these languages. Those who use NodeJS would write for it, those for Ruby too, and on... and for Drone it's all about to have a field, collecting value from Yaml key and to show this information on the interface.

[bradrydzewski]

I definitely agree with your point. The tools already exist in almost all languages to convert different file formats to a standard format (cobertura) which is the primary format used by Jenkins. Here are a few examples:

Go to cobertura https://github.com/AlekSi/gocov-xml
Python to cobertura https://pypi.python.org/pypi/nosexcover
Ruby to cobertura http://www.ruby-doc.org/gems/docs/b/buildr-gemjar-1.1.0/Buildr/Cobertura.html
Erlang to cobertura https://github.com/idubrov/covertool
Nodejs to cobertura https://github.com/gotwarlost/istanbul

I think the first step would be to handle cobertura xml files (Go has a nice xml parser). Each build would be responsible for generating the cobertura xml file.

We could come up with a custom file format for Drone, however, the cobertura xml is pretty simple and we would only really need the top node. This would be pretty easy for any language to generate:

<?xml version="1.0"?>
<coverage line-rate="0.9" branch-rate="0.75" version="1.9" timestamp="1187350905008" />

[marinho]

ok, I see. I wasn't aware on Cobertura and thought it was only a specific format for a specific language only (as coverage in Python).

Well, when the time come, I can help on it. When the design is well defined I will open a fork and implement it to the how it goes :)

Thanks for the attention!

[rgarcia]

Crazy idea: what if drone exposed a way to record simple key-val metrics about the build? Then you could produce data on all sorts of things: code coverage, benchmarks, size of executable, etc.

[rgarcia]

Inspiration: https://chromeperf.appspot.com/report

[max-koehler]

Im not sure if I oversaw it but with the implementation of coverage report it would also make sense to fail builds when the coverage degreaser by n%. I think most of us know this functionality from CI tools like Jenkins.
Any plan for these kind of integration? Could also be the first step before the integration of a complete reporting frontend.
Any idea so far how this could be integrate from an architecture POV? Would like to contribute with some more information what would be the prefered integration.

[davidak]

i would like to save different metrics for every build and get nice graphs over time.

for example to see if the tests take longer from a specific commit on.

or if the commits from one person make the code always slower.

also there could be different graphing types that can get selected for each metric.

http://www.chartjs.org/
http://www.flotcharts.org/

[bradrydzewski]

@davidak I would love this too. My biggest concern, however, is the ability for Drone to create a data structure that is generic enough to support such a diverse group of metrics and display requirements. Can we create something that will ever be good enough for all use cases? Or would we end up with something marginal at best that everyone complains about?

Can Drone report coverage as well as Coveralls? Or dependency stats as well as David DM? Or code quality as well as Code Climate?

Unfortunately all of the above are closed source. Would we be better off putting our effort into standalone, open source alternatives to the above tools? And then ensure they integrate seamlessly with Drone? Drone has a pretty engaged and motivated community, and I think we could definitely rally the community around such an effort.

I think Travis has taken the approach of delegating reporting and artifacts to third party systems and this has worked quite well. I would like to at least start with this approach and only build these capabilities directly into Drone if absolutely necessary.

---

EDIT: we have since released aircover (open source coveralls alternative) along with a drone aircover plugin. We recommend using aircover for coverage and following this pattern (external service) for any sort of reporting or artifacts.

[davidak]

we should start with simple coverage since it belongs to tests and we are running them.

as you said cobertura is a widely used format that most language support somehow.

that could be plugins that can be used somewhere else but i don't want to rely on external services.

maybe there is already code to display the result of coverage (per file, changes between build, trend, ...).

[bradrydzewski]

I think the path forward is something similar to the GitHub Status API for Drone.

We can create a Drone Status API that allows a 3rd party system to tag a commit with a very simple set of meta-data. We can surface this meta-data in the Drone user-interface along with a link to the detailed results in the 3rd party system.

We (the Drone team) can kick-off open source alternatives to these third party systems if we need to. It will honestly be easier to create these systems from scratch than cram all of this functionality into Drone. We could put engineering effort into making this simple and seamless. It could be as simple as 1 extra binary that you need to download and run.

[davidak]

sounds good. so we would have the overall coverage percentage for each build in drone and by clicking on it we get to a report in another tool that is hosted on the same server.

[bradrydzewski]

closing in favor of drone's sister project (coverage server and plugins) found at https://github.com/drone-plugins/drone-coverage as well as the proposal for the status api at #927

[Depado]

As the drone-plugins/drone-coverage repository is deprecated as it is (see drone-plugins/drone-coverage#27), should we reopen this issue to discuss the potential test coverage/badge thing ? As it seems it will be integrated in drone ?

[mvdan]

Agreed with @Depado - looking for a way to set up code coverage with Drone, and all we would really need is for it to be integrated with the web UI or to be able to have a badge.

[Graffine]

+1. We really need a way to integrate coverage rate with web UI to have a coverage badge.

[r-moiseev]

Any update on this? We need this feature too

[rcdailey]

Would be nice to hear from @bradrydzewski

[posquit0]

Would be nice to hear from @bradrydzewski