Code coverage testing

[Lestropie]

Came back to mind given I've been running some post-cmake-transition tests for #2772.

We are quite aware that our suite of tests is far from exhaustive. If we're going to the effort of performing more comprehensive tests, partly as facilitated by structural changes, and particularly if doing so has exposed a number of previously unknown bugs, it might be worth integrating quantification of coverage, both for C++ and Python, and using those holes to target expansion of the test lists. It would also push us to be more diligent in adding tests for both bug rectification and addition of new features.

[daljit46]

Code coverage analysis (for C++) is possible using the appropriate compiler flags.
For example (after adding include(CTest) at the bottom of the main CMakeLists.txt):

- cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_CXX_FLAGS="-fprofile-arcs -ftest-coverage" -DCOVERAGE_EXTRA_FLAGS="-l -p"
#last option is needed because gcov (gcc coverage tool) has issues with same file names in different folders
- ctest -R bin -T test -T coverage

produces the following output:

   Site: fedora
   Build name: Linux-c++
Performing coverage
   Processing coverage (each . represents one file):
    .................................................. processed: 50 out of 260
    .................................................. processed: 100 out of 260
    .................................................. processed: 150 out of 260
    .................................................. processed: 200 out of 260
    .................................................. processed: 250 out of 260
    ..........
   Accumulating results (each . represents one file):
    .................................................. processed: 50 out of 524
    .................................................. processed: 100 out of 524
    .................................................. processed: 150 out of 524
    .................................................. processed: 200 out of 524
    .................................................. processed: 250 out of 524
    .................................................. processed: 300 out of 524
    .................................................. processed: 350 out of 524
    .................................................. processed: 400 out of 524
    .................................................. processed: 450 out of 524
    .................................................. processed: 500 out of 524
    ........................
        Covered LOC:         25539
        Not covered LOC:     18605
        Total LOC:           44144
        Percentage Coverage: 57.85%

Note that it's important to compile in Debug mode since the compiler may optimise out some code otherwise (running the test may take a while because of this). So doing this with Github Actions' dual core machines is probably going to take a while (edit: actually just tested it and it takes about 2 hours so should be possible).

[Lestropie]

• Over and above the percentage coverage, does it provide the ability to navigate what files / lines were / were not covered? We'd want that additional info to drive deriving more tests.
• Could this initially be done just with the weekly tests on dev as per discussion in Add periodic checks with sanitizers #2763? Maybe only after investing some effort in improving overall coverage would we be interested in whether or not any individual PR modifies that value.

[daljit46]

1. Yes, that's possible. You can use gcovr (GCC Coverage Reporting Tool) to generate statistics per file. All you need to do is run gcovr in the source directory after the steps outline above. In fact, we can even generate a static website to explore these stats. Here is an example https://daljit46.github.io/mrtrix3-coverage using Github Pages that shows the coverage for a test build.
2. I guess yes. It depends how much priority we want to give to code coverage analysis (IMO it's a somewhat useful but certainly not critical metric).

[Lestropie]

1. Sweeeeet. That will definitely be useful. Would like to plug those holes at some point.
2. Agreed. It's something I see on home pages a lot, and I suppose that if one invests a lot of effort improving coverage you'd want to advertise something that you're proud of, but it's not critical. Conversely I think we should try to be more diligent in implementing tests alongside any bug fixes; so for instance we could expect that any bug fixes to master have a corresponding new test, and therefore coverage would need to increase. Not sure if it needs to be this systematic though.