Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code coverage tool anytime soon? #1157

Open
sardaukar opened this issue Aug 12, 2015 · 19 comments
Open

Code coverage tool anytime soon? #1157

sardaukar opened this issue Aug 12, 2015 · 19 comments

Comments

@sardaukar
Copy link

Is it a priority? I assume Crystal needs to provide some sort of hook for a tool like this to work, right?

@sardaukar sardaukar changed the title Coverage tool anytime soon? Code coverage tool anytime soon? Aug 12, 2015
@asterite
Copy link
Member

I believe something like what Go does (source rewrite) can be done in Crystal too.

@bcardiff
Copy link
Member

I played a bit in https://github.com/bcardiff/afix making some instrumentation of reachable code and tracing coverage over some statements (ints assignments). There I change the actual source code, but would not be that hard to hook after the parsing and add some instrumentation.

@waj
Copy link
Member

waj commented Aug 14, 2015

There is also this support from LLVM: http://llvm.org/docs/CoverageMappingFormat.html

@akzhan
Copy link
Contributor

akzhan commented May 17, 2017

Just mention about https://github.com/SlatherOrg/slather as example of publishing llvm-cov results to coverage web sites (like coverall, codecov).

@RX14
Copy link
Contributor

RX14 commented May 17, 2017

I just looked at what rust uses for coverage, and found kcov, which looks to be a tool which can generate coverage reports from arbitrary precompiled executables just using their debug information. It seems to work for crystal pretty reliably with no modifications (although it confirmed my suspicion that crystal's debug info needs lots more work from the amount of coverage in doc comments). I'll probably end up integrating it into crystal's jenkins build, which I hope will accelerate getting crystal perfectly compatible with kcov.

@watzon
Copy link
Contributor

watzon commented Feb 25, 2018

Any progress on this? Code coverage would be great.

@RX14
Copy link
Contributor

RX14 commented Feb 25, 2018

@watzon the kcov solution seems pretty good. The problem is that somehow LLVM garbles crystal's debug info. That bug needs to be worked around and then kcov would be essentially close to perfect.

@watzon
Copy link
Contributor

watzon commented Feb 25, 2018

@RX14 oh awesome. Is there an issue for that?

@RX14
Copy link
Contributor

RX14 commented Mar 1, 2018

#5567 and #4718 are relevant.

@anykeyh
Copy link

anykeyh commented Apr 28, 2018

I take opportunity of this thread to advertise since it's exactly what I'm currently working on:

https://github.com/anykeyh/crystal-coverage

Cannot promise it will not fail on your code (still proof of concept) but in the ideal case, it will generate a proper coverage report 👍

@asterite
Copy link
Member

Just a note that I tried such approach in the past and compilation times went to the roof, so I literally couldn't reach the end of compilation to test it in a real project other than a hello word (or was it in a hello world?)

At least my approach was to add some method call at the beginning and end6of each method. If that's your case, don't hold your hopes tok high.

The proper way to do this is using LLVM's coverage feature, which I think @RX14 has done in another issue (as a proof of concept).

@anykeyh
Copy link

anykeyh commented Apr 28, 2018

@asterite, my approach is basically the same yes. I've tried using LLVM coverage feature and kcov but I wasn't able to reproduce (I don't remember why, was few months ago).

Edit: Basically I use a slightly other approach and try to flavor every relevant lines of code with a counter of access (inspired by this: https://blog.golang.org/cover)

So far I've tried the shard on projects with around 3000 lines of code. I don't know if the compilation time is linear or not to number of lines of code, but as of now I don't see any issue with compile time.

What would increase compilation time in the case of bigger projects? Basically it just flavors the code with new basic operations, with maybe an overhead of 30% of lines of code.

Yet it doesn't take advantage of any caching at recompilation, even if it's a feature which should be implemented easily.

@asterite
Copy link
Member

@anykeyh Cool! Maybe either the compiler's speed improved, relative to when I tried to implement it, or we are doing it in a different way (that is, I was doing it in a dumber way 😅).

If that works, and in the end it isn't too much code, we could eventually incorporate it as a built-in feature of the compiler, if you don't mind.

@anykeyh
Copy link

anykeyh commented Apr 28, 2018

Would be great indeed ! Let's make it fully work first, see if there's no hidden flaws, but I love the idea of crystal --cover mycode.cr or something like that :)

@RX14
Copy link
Contributor

RX14 commented Apr 28, 2018

@anykeyh the ideal project to test it on is of course the crystal compiler specs. If you can run it on the compiler and stdlib spec suite, you can be sure you have most compilation errors sorted - and it's scalable.

@paulocoghi
Copy link

paulocoghi commented Jun 7, 2021

It would be interesting if the coverage report is able to evaluate all the different logical possibilities and ramifications, as Jest does:
(informing the coverage of "statements", "branch", "functions", "lines" and their logical ramifications)

----------|---------|----------|---------|---------|-------------------
File      | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
----------|---------|----------|---------|---------|-------------------
All files |     100 |      100 |     100 |     100 |                   
 index.js |     100 |      100 |     100 |     100 |                   
----------|---------|----------|---------|---------|-------------------
Test Suites: 1 passed, 1 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        3.735 s, estimated 5 s
Ran all test suites.

@caspiano
Copy link
Collaborator

I've been using crystal-kcov over the last few days!
It's great, and doesn't break on projects with lots of macro code - something I've found crystal-coverage has a tendency of doing

@paulocoghi
Copy link

paulocoghi commented Nov 15, 2021

I remember reading some of @asterite's comments explaining that Crystal should still provide some extra logs/debug info to be able to be fully covered.

Is this the case with crystal-kcov?

@Blacksmoke16
Copy link
Member

Blacksmoke16 commented Mar 2, 2023

Ideally Crystal could probably just leverage https://llvm.org/docs/CoverageMappingFormat.html#high-level-overview to make use of existing LLVM features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests