New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Code coverage tool anytime soon? #1157
Comments
I believe something like what Go does (source rewrite) can be done in Crystal too. |
I played a bit in https://github.com/bcardiff/afix making some instrumentation of reachable code and tracing coverage over some statements (ints assignments). There I change the actual source code, but would not be that hard to hook after the parsing and add some instrumentation. |
There is also this support from LLVM: http://llvm.org/docs/CoverageMappingFormat.html |
Just mention about https://github.com/SlatherOrg/slather as example of publishing |
I just looked at what rust uses for coverage, and found |
Any progress on this? Code coverage would be great. |
@watzon the kcov solution seems pretty good. The problem is that somehow LLVM garbles crystal's debug info. That bug needs to be worked around and then kcov would be essentially close to perfect. |
@RX14 oh awesome. Is there an issue for that? |
I take opportunity of this thread to advertise since it's exactly what I'm currently working on: https://github.com/anykeyh/crystal-coverage Cannot promise it will not fail on your code (still proof of concept) but in the ideal case, it will generate a proper coverage report 👍 |
Just a note that I tried such approach in the past and compilation times went to the roof, so I literally couldn't reach the end of compilation to test it in a real project other than a hello word (or was it in a hello world?) At least my approach was to add some method call at the beginning and end6of each method. If that's your case, don't hold your hopes tok high. The proper way to do this is using LLVM's coverage feature, which I think @RX14 has done in another issue (as a proof of concept). |
@asterite, my approach is basically the same yes. I've tried using LLVM coverage feature and kcov but I wasn't able to reproduce (I don't remember why, was few months ago). Edit: Basically I use a slightly other approach and try to flavor every relevant lines of code with a counter of access (inspired by this: https://blog.golang.org/cover) So far I've tried the shard on projects with around 3000 lines of code. I don't know if the compilation time is linear or not to number of lines of code, but as of now I don't see any issue with compile time. What would increase compilation time in the case of bigger projects? Basically it just flavors the code with new basic operations, with maybe an overhead of 30% of lines of code. Yet it doesn't take advantage of any caching at recompilation, even if it's a feature which should be implemented easily. |
@anykeyh Cool! Maybe either the compiler's speed improved, relative to when I tried to implement it, or we are doing it in a different way (that is, I was doing it in a dumber way 😅). If that works, and in the end it isn't too much code, we could eventually incorporate it as a built-in feature of the compiler, if you don't mind. |
Would be great indeed ! Let's make it fully work first, see if there's no hidden flaws, but I love the idea of |
@anykeyh the ideal project to test it on is of course the crystal compiler specs. If you can run it on the compiler and stdlib spec suite, you can be sure you have most compilation errors sorted - and it's scalable. |
It would be interesting if the coverage report is able to evaluate all the different logical possibilities and ramifications, as Jest does:
|
I've been using crystal-kcov over the last few days! |
I remember reading some of @asterite's comments explaining that Crystal should still provide some extra logs/debug info to be able to be fully covered. Is this the case with |
Ideally Crystal could probably just leverage https://llvm.org/docs/CoverageMappingFormat.html#high-level-overview to make use of existing LLVM features. |
Is it a priority? I assume Crystal needs to provide some sort of hook for a tool like this to work, right?
The text was updated successfully, but these errors were encountered: