Is the reported coverage fair?

[lindig]

Bisect_ppx has the notion of code points for which it collects runtime data. Hitting some of these code points in OCaml has quite a different meaning as when hitting a statement in a procedural language. For example, hitting a sequence point implies that a list or sequence of statements will be evaluated that can easily span several lines. I therefore believe that a fair measure for coverage is what fraction of all code points are executed - independent of line numbers.

I understand that cveralls wants line numbers but in general there will be more line numbers than code points. This means, it is almost impossible to achieve high coverage with data reported by ocveralls. In addition, ocveralls reports a line with several code points on only as hit when all its points have been hit - exaggerating the problem.

[aantron]

@lindig I'm under the impression that the coveralls coverage number is the fraction of instrumented lines (not all lines) that have been visited. So, actually, I think the number of instrumented lines is less than the number of code points.

The numbers reported by coveralls and Bisect_ppx do differ (as expected) due to counting lines and points, respectively, but I've found that the difference is not much in practice – just a few percent. Have you encountered a pathological case?

I think this is the whole problem:

ocveralls reports a line with several code points on only as hit when all its points have been hit - exaggerating the problem.

[lindig]

I think you are right that there is a distinction between covered and uncovered lines - which makes it fair again. I had overlooked the null values in the JSON data sent to cverall:

Acceptable values for the array is an integer representing the number of times that line is covered, or null representing a line that is not relevant to coverage (such as whitespace or a comment).

How to treat several points on a line best, I don't know either. But I'd no longer consider this an issue - feel free to close it.

[sagotch]

About the problem of partially hit lines, the problem is on the coveralls.io side, even if an issue about this has been open two years ago (lemurheavy/coveralls-public#216).

So the only solution is probably to switch to another service...