Coverage doesn't work if the test suite doesn't depend on the library #3682
Comments
|
/cc @ttuegel |
|
Oddly enough, EDIT: The difference would be that we're not passing |
|
Got it. The immediate cause for failure is the The actual cause for failure is hspec and co are being compiled with coverage enabled. Why? Because you did There's probably something very wonky going on with the implicit assumptions Setup makes which should be fixed but I haven't thought about it carefully enough. |
|
Unfortunately that does not solve the problem. I did exactly the steps you said (including https://travis-ci.org/snapframework/snap-core/jobs/150434976 And here is the diff between those commits |
|
Oh ho ho, what a confusing ticket. It turns out that the error I pasted above was NOT the error you were trying to report (rereading your initial report, it looks like you did NOT get any error or non-zero exit code, just nothing in the coverage. This is why it's good to always post logs when you have them.) I looked at my now succeeding hpc output and it is indeed empty. And I also know why this is the case: Look at the invocation of So why is it excluding your library files? Well, look at your Cabal file, you didn't declare a dependency on the library; you're using the hack (fwiw, cabal-install uses this hack too, so not such a hack) where you put Workarounds: call hpc yourself without the exclude flags. Or make the test suite depend on the library and remove the My recommendation on the Cabal side is we should just remove the |
This was requested by someone at some point. (I don't recall exactly; but I'm exceptionally lazy and I never would have done it out-of-the-blue 😉) I agree that it seems overengineered. Also, doesn't |
|
@ttuegel are you planning on fixing this? |
|
Ok, I think I understand what is going on now. I have a couple more comments. First, when I make the test suite depend on the library it works almost as expected. However, there is one little frustrating detail...a random hash gets prepended to the module names:
This is fine for my own consumption, but it's a little annoying for this one which I am auto-generating from my Snap build and publishing on the Snap website. This is fairly minor and I might not even mention except that we already have to consider the issue that the approach of having the test suite depend on the library does not work in all situations. As you mention, you use the I also spent a lot of time yesterday fighting with this stuff for the snap release. That case was a little different because we are using new-build. In this issue where I'm not using new-build it was already pretty hard to discover the right hpc incantation. In the new-build situation it's even worse because cabal doesn't give you any examples of how to invoke hpc. I definitely appreciate the spirit of what cabal is trying to do about automatically making sure the test suite files don't show in the coverage reports. Here's the solution I ended up with for the map-syntax package that I was using in this ticket: https://github.com/mightybyte/map-syntax/blob/master/runCoverage.sh It would certainly be nice to not have to manually construct the list of excludes, but it's no good if it doesn't work in all situations. My main concern with all this is making information more discoverable so it's easier for me (and other people in the future) to get something analogous to that runCoverage script into its final form. The biggest problem there was figuring out what the |
Is this related to https://ghc.haskell.org/trac/ghc/ticket/10952 ? I guess this is a different aesthetic issue. The source of the issue might be hpc or GHC or Cabal, hard to say. CC @mgsloan who has also looked at this.
This is exactly what convenience/internal libraries are for, which are coming next Cabal release! I don't know what your Cabal support window looks like but eventually you will be able to put your internal code in an internal, non-published library component, and then have your public library as well as your test suites depend on it. So yes it's a hack, but a justifiable one today, and perhaps less justifiable three years from now. (If you have a three GHC release window...)
Well, it is not surprising you had difficulty, since it simply not supported at all right now (just like how new-run, new-test, etc are not supported). I'll file another ticket about getting hpc to work with new-build. There are a lot of bugs to fix and features to implement (just look at the nix-local-build tag on the issues tracker); we could definitely use more hands. (Submit a PR and you'll get commit bits #3567 ;)
Having never used hpc before, I have no idea what this should look like. Suggestions welcome :) |
ezyang
changed the title
Awesome, that sounds great! I'll have to think about the compatibility thing, but it's nice to know this is in the works.
Well, you know what hpc command is generated by the existing |
|
Hi there, It seems I have the very same problem just with a little tweak: So i have something alike: running The full project can be found here Is there any way I can help with this? |
|
This is related to #5213, but I doubt the PR that fixes #5213 will fix that (if it's still buggy --- I haven't tested). @lapplislazuli: I guess you can place all the modules from Hasculator into an internal library and only have a tiny wrapper as a test. |
|
@lapplislazuli: Actually, you'd probably get "cabal: Error: test coverage is only supported for packages with a library" so that's not a valid workaround. I tried with newest development cabal and GHC 9.0.1. However, you may also turn on HPC, build exe, run it and then manually create HPC report and HPC makrup. At least that's what I was doing some years ago, though now my scripts are bit-rotten. |
|
BTW, in #7250 the |
To reproduce this, first clone this repo:
Then the following inside the map-syntax repo generates nice HPC test coverage reports:
This appears to generate HPC test coverage reports, but they are actually empty:
In this case,
cabal testis supposed to generate the coverage reports for you, so the lasthpcline from the other example should not be needed. The only difference between the two branches is aghc-options: -fhpcline in the cabal file.The biggest problem here is that I actually can't use the hpc-works approach because
cabal sdistsays that I should not use the-fhpcoption inside my .cabal file and that hackage will reject my package.One other thing that would be good to keep in mind here is that
hpcallows you to tweak things with a variety of command line flags like --exclude, --include, etc. Whatever solution cabal gives us should still make it possible to specify your own flags. Maybe this is as simple as putting the coverage data in a predictable place so you can runhpcmanually. But it is still probably something that we need to think about if we're going to continue outlawing-fhpcin .cabal files.This is with cabal-install-1.25.0.0. I think built from commit a92bfc8
The text was updated successfully, but these errors were encountered: