Code coverage tool anytime soon?

[sardaukar]

Is it a priority? I assume Crystal needs to provide some sort of hook for a tool like this to work, right?

[asterite]

I believe something like what Go does (source rewrite) can be done in Crystal too.

[bcardiff]

I played a bit in https://github.com/bcardiff/afix making some instrumentation of reachable code and tracing coverage over some statements (ints assignments). There I change the actual source code, but would not be that hard to hook after the parsing and add some instrumentation.

[waj]

There is also this support from LLVM: http://llvm.org/docs/CoverageMappingFormat.html

[akzhan]

Just mention about https://github.com/SlatherOrg/slather as example of publishing llvm-cov results to coverage web sites (like coverall, codecov).

[RX14]

I just looked at what rust uses for coverage, and found kcov, which looks to be a tool which can generate coverage reports from arbitrary precompiled executables just using their debug information. It seems to work for crystal pretty reliably with no modifications (although it confirmed my suspicion that crystal's debug info needs lots more work from the amount of coverage in doc comments). I'll probably end up integrating it into crystal's jenkins build, which I hope will accelerate getting crystal perfectly compatible with kcov.

[watzon]

Any progress on this? Code coverage would be great.

[RX14]

@watzon the kcov solution seems pretty good. The problem is that somehow LLVM garbles crystal's debug info. That bug needs to be worked around and then kcov would be essentially close to perfect.

[watzon]

@RX14 oh awesome. Is there an issue for that?

[RX14]

#5567 and #4718 are relevant.

[anykeyh]

I take opportunity of this thread to advertise since it's exactly what I'm currently working on:

https://github.com/anykeyh/crystal-coverage

Cannot promise it will not fail on your code (still proof of concept) but in the ideal case, it will generate a proper coverage report 👍

[asterite]

Just a note that I tried such approach in the past and compilation times went to the roof, so I literally couldn't reach the end of compilation to test it in a real project other than a hello word (or was it in a hello world?)

At least my approach was to add some method call at the beginning and end6of each method. If that's your case, don't hold your hopes tok high.

The proper way to do this is using LLVM's coverage feature, which I think @RX14 has done in another issue (as a proof of concept).

[anykeyh]

@asterite, my approach is basically the same yes. I've tried using LLVM coverage feature and kcov but I wasn't able to reproduce (I don't remember why, was few months ago).

Edit: Basically I use a slightly other approach and try to flavor every relevant lines of code with a counter of access (inspired by this: https://blog.golang.org/cover)

So far I've tried the shard on projects with around 3000 lines of code. I don't know if the compilation time is linear or not to number of lines of code, but as of now I don't see any issue with compile time.

What would increase compilation time in the case of bigger projects? Basically it just flavors the code with new basic operations, with maybe an overhead of 30% of lines of code.

Yet it doesn't take advantage of any caching at recompilation, even if it's a feature which should be implemented easily.

[asterite]

@anykeyh Cool! Maybe either the compiler's speed improved, relative to when I tried to implement it, or we are doing it in a different way (that is, I was doing it in a dumber way 😅).

If that works, and in the end it isn't too much code, we could eventually incorporate it as a built-in feature of the compiler, if you don't mind.

[anykeyh]

Would be great indeed ! Let's make it fully work first, see if there's no hidden flaws, but I love the idea of crystal --cover mycode.cr or something like that :)

[RX14]

@anykeyh the ideal project to test it on is of course the crystal compiler specs. If you can run it on the compiler and stdlib spec suite, you can be sure you have most compilation errors sorted - and it's scalable.

[paulocoghi]

It would be interesting if the coverage report is able to evaluate all the different logical possibilities and ramifications, as Jest does:
(informing the coverage of "statements", "branch", "functions", "lines" and their logical ramifications)

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |     100 |      100 |     100 |     100 |                   
 index.js |     100 |      100 |     100 |     100 |                   
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 passed, 1 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        3.735 s, estimated 5 s
Ran all test suites.

[caspiano]

I've been using crystal-kcov over the last few days!
It's great, and doesn't break on projects with lots of macro code - something I've found crystal-coverage has a tendency of doing

[paulocoghi]

I remember reading some of @asterite's comments explaining that Crystal should still provide some extra logs/debug info to be able to be fully covered.

Is this the case with crystal-kcov?

[Blacksmoke16]

Ideally Crystal could probably just leverage https://llvm.org/docs/CoverageMappingFormat.html#high-level-overview to make use of existing LLVM features.