Recreating the test coverage results reported on Hackage

[ivanperez-keera]

When Hackage runs the tests of this project, it provides coverage results for each of the libraries.

While I can run the tests with coverage enabled, I am failing to reproduce the results that Hackage shows. The numbers I obtain are always off by a bit. I'm more comfortable with cabal v1-* commands, which are considered outdated, and even running the tests with coverage is hard (I cannot always get HPC to report data regarding new modules and new tests, and I just don't know why).

Is there a way to create a Dockerfile that will run the tests of one library, copilot-core, and produce results that are consistent with what Hackage is reporting (with whatever version of Cabal and whatever commands are needed). As a starting point, there are quite a few Dockerfiles in the comments of the PRs: https://github.com/Copilot-Language/copilot/pulls?q=is%3Apr+is%3Aclosed

[ivanperez-keera]

For reference: https://www.reddit.com/r/haskell/comments/1oydrv1/comment/np6vhzm/?context=1

[RyanGlScott]

I wish this information were easy to reproduce locally, but unfortunately, it's not quite so. I ended up reading hackage-server's source code in order to reverse engineer how to do this.

As a running example, let's pick copilot-core-4.6, whose Hackage page contains a code coverage badge that shows 53%. How was that 53% number obtained? On its own, that question can be answered relatively easily: go to the reports page for copilot-core-4.6, click the most recent build report, and see the following coverage-related table displayed:

expressions	49% (165/334)
        guards	9% (1/11)
boolean conditions	100% (0/0)
        qualifiers	100% (0/0)
alternatives	79% (38/48)
local declarations	61% (8/13)
top-level declarations	61% (41/67)

From this information, the 53% figure can be computed by taking the number of things covered (i.e., the sum of the numbers to the left of the / sign) divided by the number of total things in the program (i.e., the sum of the numbers to the right of the / sign) times 100%. That is, (165+1+0+0+38+8+41) / (334+11+0+0+48+13+67) * 100%, which rounds to 53%.

The real question is: how does hackage-server obtain this table? To do that, you have to perform a series of hpc-related commands. First, you have to run the copilot-core-4.6 test suite with coverage enabled:

$ cabal test --enable-coverage

This will generate a .tix file, as well as a mix directory. You then need to run hpc report on these files and directories, and importantly, you need to ensure that only the code from the copilot-core-4.6 library itself is included in the report. Make sure not to include any code from the test suite, as this code does not appear to be included in the coverage table that hackage-server reports.

On my machine, I accomplished this by running the following command (using cabal-install version 3.16.0.0):

$ hpc report \
    dist-newstyle/build/aarch64-osx/ghc-9.4.8/copilot-core-4.6/t/unit-tests/hpc/vanilla/tix/unit-tests.tix \
    --hpcdir=dist-newstyle/build/aarch64-osx/ghc-9.4.8/copilot-core-4.6/build/extra-compilation-artifacts/hpc/vanilla/mix \
    --include="Copilot.Core.Type.Array" --include="Copilot.Core.Type" # Only include code from library itself

 49% expressions used (165/334)
  9% boolean coverage (1/11)
       9% guards (1/11), 3 always True, 7 unevaluated
     100% 'if' conditions (0/0)
     100% qualifiers (0/0)
 79% alternatives used (38/48)
 61% local declarations used (8/13)
 61% top-level declarations used (41/67)

Note that I am explicitly enumerating the modules that I want to include in the coverage report by listing each of copilot-core's modules with --include flags. Also note that the exact locations of the .tix file and mix directory may depend on the specifics of your machine and which version of cabal you are using, but they should look roughly like what I did above.

As far as how to replicate this in a Dockerfile... unfortunately, I think you'll have to do some post-processing of the hpc report output in order to do this. (See the relevant hackage-server code that scrapes this output.)

[RyanGlScott]

Interestingly, I don't have to bother with --include when using v1-style commands.

Here is a Dockerfile that gives you the hpc report output, at the very least:

FROM ubuntu:focal

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update

RUN apt-get install --yes \
      libz-dev \
      git \
      curl \
      gcc \
      g++ \
      make \
      libgmp3-dev  \
      pkg-config \
      z3

RUN mkdir -p $HOME/.ghcup/bin
RUN curl https://downloads.haskell.org/~ghcup/0.1.40.0/x86_64-linux-ghcup-0.1.40.0 -o $HOME/.ghcup/bin/ghcup
RUN chmod a+x $HOME/.ghcup/bin/ghcup
ENV PATH=$PATH:/root/.ghcup/bin/
ENV PATH=$PATH:/root/.cabal/bin/

SHELL ["/bin/bash", "-c"]

RUN ghcup install ghc 9.10.1
RUN ghcup install cabal 3.2
RUN ghcup set ghc 9.10.1
RUN cabal update

SHELL ["/bin/bash", "-c"]
CMD git clone https://github.com/Copilot-Language/copilot && cd copilot/copilot-core && git checkout v4.6 \
  && cabal v1-sandbox init \
  && cabal v1-install alex happy --constraint='happy <= 2' \
  && cabal v1-install --enable-tests --only-dependencies \
  && cabal v1-configure --enable-library-coverage \
  && cabal v1-test \
  && hpc report unit-tests.tix --hpcdir=$(ls -d $(find dist/ -iname "mix" -type d)/* | head -1)

Scraping the output is left as future work :)

[ivanperez-keera]

Thanks! Can we modify this to use a mounted volume rather than cloning Copilot in the CMD?

What happens if one runs cabal v1-test twice? I'd like to make sure cabal discards or overwrites the files, rather than accumulating the results, so that the metrics obtained are fresh. Additionally, when that happens, I'd like to make sure that new modules being tested are picked up.

Also, you may know this already, but you can:

git clone -b v4.6 https://github.com/Copilot-Language/copilot

to switch to a tag or branch. In Copilot, it doesn't make a big difference because we don't have submodules but, when you clone with --recursive, it simplifies the process to add the -b to the clone operation (specifically, do don't need to do the submodules init or update submodules recursively afterwards).

[RyanGlScott]

Can we modify this to use a mounted volume rather than cloning Copilot in the CMD?

Sure, feel free to modify this as you see fit. This is just a simple proof of concept inspired by an already existing Dockerfile (located here).

What happens if one runs cabal v1-test twice?

Unfortunately, v1-test appears not to overwrite the contents of unit-tests.tix, so if you want to re-run coverage, you'll need to rm unit-tests.txt and then run cabal v1-test again.

[ivanperez-keera]

Are there any other files (e.g., mix files) that we'd need to remove?

[RyanGlScott]

I didn't appear to need to remove any of the mix files when I tried this locally.

[ivanperez-keera]

A while back I was trying to get this to work and I found this code in Hackage, which is how it calculates the average shown on the main page for each package:

        getAvgCovg :: Maybe BR.BuildCovg -> (Bool, String, Int)                                                                                                                                                                                                                           
        getAvgCovg Nothing = (False, "", 100)                                    
        getAvgCovg (Just c) = do                                                 
              let l = [                                                          
                        BR.expressions c                                         
                      , BR.guards (BR.boolean c)                                 
                      , BR.ifConditions (BR.boolean c)                           
                      , BR.qualifiers (BR.boolean c)                             
                      , BR.alternatives c                                        
                      , BR.localDeclarations c                                   
                      , BR.topLevel c                                            
                      ]                                                          
                  (used,total) = foldl (\(a,b) (x, y) -> (a+x, b+y)) (0,0) l     
                  per | total <= 0 = 100                                         
                      | otherwise = (used*100) `div` total                       
              if per > 66                                                        
                then (True, "brightgreen", per)                                  
                else if per > 33                                                 
                  then (True, "yellowgreen", per)                                
                  else (True, "red", per)