Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Code coverage tool anytime soon? #1157

Open
sardaukar opened this Issue Aug 12, 2015 · 15 comments

Comments

Projects
None yet
9 participants
@sardaukar
Copy link

sardaukar commented Aug 12, 2015

Is it a priority? I assume Crystal needs to provide some sort of hook for a tool like this to work, right?

@sardaukar sardaukar changed the title Coverage tool anytime soon? Code coverage tool anytime soon? Aug 12, 2015

@asterite

This comment has been minimized.

Copy link
Member

asterite commented Aug 12, 2015

I believe something like what Go does (source rewrite) can be done in Crystal too.

@bcardiff

This comment has been minimized.

Copy link
Member

bcardiff commented Aug 12, 2015

I played a bit in https://github.com/bcardiff/afix making some instrumentation of reachable code and tracing coverage over some statements (ints assignments). There I change the actual source code, but would not be that hard to hook after the parsing and add some instrumentation.

@waj

This comment has been minimized.

Copy link
Member

waj commented Aug 14, 2015

There is also this support from LLVM: http://llvm.org/docs/CoverageMappingFormat.html

@akzhan

This comment has been minimized.

Copy link
Contributor

akzhan commented May 17, 2017

Just mention about https://github.com/SlatherOrg/slather as example of publishing llvm-cov results to coverage web sites (like coverall, codecov).

@RX14

This comment has been minimized.

Copy link
Member

RX14 commented May 17, 2017

I just looked at what rust uses for coverage, and found kcov, which looks to be a tool which can generate coverage reports from arbitrary precompiled executables just using their debug information. It seems to work for crystal pretty reliably with no modifications (although it confirmed my suspicion that crystal's debug info needs lots more work from the amount of coverage in doc comments). I'll probably end up integrating it into crystal's jenkins build, which I hope will accelerate getting crystal perfectly compatible with kcov.

@watzon

This comment has been minimized.

Copy link

watzon commented Feb 25, 2018

Any progress on this? Code coverage would be great.

@RX14

This comment has been minimized.

Copy link
Member

RX14 commented Feb 25, 2018

@watzon the kcov solution seems pretty good. The problem is that somehow LLVM garbles crystal's debug info. That bug needs to be worked around and then kcov would be essentially close to perfect.

@watzon

This comment has been minimized.

Copy link

watzon commented Feb 25, 2018

@RX14 oh awesome. Is there an issue for that?

@RX14

This comment has been minimized.

Copy link
Member

RX14 commented Mar 1, 2018

#5567 and #4718 are relevant.

@anykeyh

This comment has been minimized.

Copy link

anykeyh commented Apr 28, 2018

I take opportunity of this thread to advertise since it's exactly what I'm currently working on:

https://github.com/anykeyh/crystal-coverage

Cannot promise it will not fail on your code (still proof of concept) but in the ideal case, it will generate a proper coverage report 👍

@asterite

This comment has been minimized.

Copy link
Member

asterite commented Apr 28, 2018

Just a note that I tried such approach in the past and compilation times went to the roof, so I literally couldn't reach the end of compilation to test it in a real project other than a hello word (or was it in a hello world?)

At least my approach was to add some method call at the beginning and end6of each method. If that's your case, don't hold your hopes tok high.

The proper way to do this is using LLVM's coverage feature, which I think @RX14 has done in another issue (as a proof of concept).

@anykeyh

This comment has been minimized.

Copy link

anykeyh commented Apr 28, 2018

@asterite, my approach is basically the same yes. I've tried using LLVM coverage feature and kcov but I wasn't able to reproduce (I don't remember why, was few months ago).

Edit: Basically I use a slightly other approach and try to flavor every relevant lines of code with a counter of access (inspired by this: https://blog.golang.org/cover)

So far I've tried the shard on projects with around 3000 lines of code. I don't know if the compilation time is linear or not to number of lines of code, but as of now I don't see any issue with compile time.

What would increase compilation time in the case of bigger projects? Basically it just flavors the code with new basic operations, with maybe an overhead of 30% of lines of code.

Yet it doesn't take advantage of any caching at recompilation, even if it's a feature which should be implemented easily.

@asterite

This comment has been minimized.

Copy link
Member

asterite commented Apr 28, 2018

@anykeyh Cool! Maybe either the compiler's speed improved, relative to when I tried to implement it, or we are doing it in a different way (that is, I was doing it in a dumber way 😅).

If that works, and in the end it isn't too much code, we could eventually incorporate it as a built-in feature of the compiler, if you don't mind.

@anykeyh

This comment has been minimized.

Copy link

anykeyh commented Apr 28, 2018

Would be great indeed ! Let's make it fully work first, see if there's no hidden flaws, but I love the idea of crystal --cover mycode.cr or something like that :)

@RX14

This comment has been minimized.

Copy link
Member

RX14 commented Apr 28, 2018

@anykeyh the ideal project to test it on is of course the crystal compiler specs. If you can run it on the compiler and stdlib spec suite, you can be sure you have most compilation errors sorted - and it's scalable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.