Excluding dependencies in code coverage reports

[mscofield0]

I am using CPM for managing dependencies in my project and since it is source-based (downloads the source and then builds it), llvm-cov finds and includes it in the code coverage report.

I am using the caching feature of CPM so everything is in a .cache/CPM directory.

What do you suggest I do?

[StableCoder]

Both the add_code_coverage_all_targets and target_code_coverage functions should have the ability to exclude files from being profiled with glob patterns:

add_code_coverage_all_targets(EXCLUDE .*/test/.*)
*or*
target_code_coverage(test_lib EXCLUDE .*/test/.*)

If you're using the add_code_coverage, it adds profiling to everything after the point where it's included, so something along the lines of:

*cpm_builds*

add_code_coverage()

*add_profiled_targets*

could work.

[mscofield0]

The following doesn't work...
I want to exclude the test directory and the dependencies stored at the CPM_SOURCE_CACHE. The current working directory of the CMakeLists.txt is test.

if(DEFINED ENV{CPM_SOURCE_CACHE})
  add_code_coverage_all_targets(EXCLUDE $ENV{CPM_SOURCE_CACHE}/.*)
else()
  add_code_coverage_all_targets(EXCLUDE build/.*)
endif()

target_code_coverage(Greeter)
target_code_coverage(GreeterTests AUTO .*)

[StableCoder]

Ah, I see.

To maintain flexibility between individual targets and the `all-type targets, they all maintain their own exclusion lists.

So in this case, I'm pretty sure that, with the code above, running ccov-all or similar, the results would have the CPM items excluded, and the individual ccov-Greeter and ccov-GreeterTests would still have them included.

So to resolve here, I believe you just have to add the same excludes to the individual target_code_coverage calls, ala:

if(DEFINED ENV{CPM_SOURCE_CACHE})
  add_code_coverage_all_targets(EXCLUDE $ENV{CPM_SOURCE_CACHE}/.* build/.*)
else()
  add_code_coverage_all_targets(EXCLUDE build/.*)
endif()

target_code_coverage(Greeter EXCLUDE $ENV{CPM_SOURCE_CACHE}/.* build/.*)
target_code_coverage(GreeterTests AUTO EXCLUDE .* $ENV{CPM_SOURCE_CACHE}/.*)

[mscofield0]

I actually had an error in the above command, I didn't type EXCLUDE, but it doesn't seem to make a difference:

if(DEFINED ENV{CPM_SOURCE_CACHE})
  add_code_coverage_all_targets(EXCLUDE $ENV{CPM_SOURCE_CACHE}/.* build/.*)
else()
  add_code_coverage_all_targets(EXCLUDE build/.*)
endif()


target_code_coverage(Greeter EXCLUDE $ENV{CPM_SOURCE_CACHE}/.* build/.*)
target_code_coverage(GreeterTests AUTO EXCLUDE .* $ENV{CPM_SOURCE_CACHE}/.*)

/code-coverage.cmake:447 (add_dependencies):
  The dependency target "ccov-report-GreeterTests" of target "ccov-report"
  does not exist.

[StableCoder]

Alright, new theory.

After far too long, I think that the problem is that clang/LLVM excludes via a regex pattern, and GCC/lcov uses a globbing pattern. So try out having a pattern for both and see how that plays out:

if(DEFINED ENV{CPM_SOURCE_CACHE})
  add_code_coverage_all_targets(EXCLUDE 
# For LLVM/clang
$ENV{CPM_SOURCE_CACHE}/.* build/.*
# For GCC/lcov
$ENV{CPM_SOURCE_CACHE}/* */build/*
)
else()
  add_code_coverage_all_targets(EXCLUDE build/.* */build/*)
endif()


target_code_coverage(Greeter EXCLUDE $ENV{CPM_SOURCE_CACHE}/.* build/.* $ENV{CPM_SOURCE_CACHE}/* */build/*)
target_code_coverage(GreeterTests AUTO EXCLUDE .* * $ENV{CPM_SOURCE_CACHE}/.* $ENV{CPM_SOURCE_CACHE}/*)

See if that starts removing things?

[mscofield0]

llvm-cov runs normally, but still doesn't exclude the directories we set.

And with gcc it just doesn't work (same error as before):

code-coverage.cmake:424 (add_dependencies):
  The dependency target "ccov-report-GreeterTests" of target "ccov-report"
  does not exist.

[StableCoder]

I'm having trouble reproducing this. Is there a repo/commit somewhere you can point me to with the issue?

[mscofield0]

I've made it public https://github.com/mscofield0/ModernCppStarter

[StableCoder]

Alright, I've got something that's works for me, with your repo, try it out.

# ---- code coverage ----

if(DEFINED ENV{CPM_SOURCE_CACHE})
  set(CCOV_EXCLUDES $ENV{CPM_SOURCE_CACHE}/.* $ENV{CPM_SOURCE_CACHE}/*)
endif()
set(CCOV_EXCLUDES ${CMAKE_BINARY_DIR}/.* ${CMAKE_BINARY_DIR}/*)

add_code_coverage_all_targets(EXCLUDE ${CCOV_EXCLUDES})

target_code_coverage(Greeter EXCLUDE ${CCOV_EXCLUDES})
target_code_coverage(GreeterTests AUTO EXCLUDE ${CCOV_EXCLUDES})

The biggest thing that was wrecking me was the $ENV{CPM_SOURCE_CACHE}/, which if I didn't have it defined, meant that it was equivalent to '/', ie everything on a unix filesystem.

Secondly was the build/* items, which for GCC appears in the .info files with the full path, and thus starting with just 'build' would match almost nothing. CHanging these to the CMAKE_BINARY_DIR, which is the root of the generated build files, including what CPM brings in by default, means all that would be automatically excluded.

Also, if when running the GCC ccov you have an issue with the ccov-report target, update the CPMAddPackage for this repo from aa8c42b to e07087b or newer.

[mscofield0]

It works on gcc lcov for me, but not for llvm-cov, the llvm-cov output remains the same as before.

This is the CMake code:

if(DEFINED ENV{CPM_SOURCE_CACHE})
  # file(GLOB_RECURSE CCOV_EXCLUDES CONFIGURE_DEPENDS $ENV{CPM_SOURCE_CACHE}/*)
  set(CCOV_EXCLUDES $ENV{CPM_SOURCE_CACHE}/.* $ENV{CPM_SOURCE_CACHE}/*)
endif()
# file(GLOB_RECURSE CCOV_EXCLUDES CONFIGURE_DEPENDS ${CMAKE_BINARY_DIR}/*)
set(CCOV_EXCLUDES ${CMAKE_BINARY_DIR}/.* ${CMAKE_BINARY_DIR}/*)

add_code_coverage_all_targets(EXCLUDE ${CCOV_EXCLUDES})

target_code_coverage(Greeter EXCLUDE ${CCOV_EXCLUDES})
target_code_coverage(GreeterTests AUTO EXCLUDE ${CCOV_EXCLUDES})

[mscofield0]

Bumping the issue