Coverage builds not taking into account all testing files

[Blast545]

Bug report

Required Info:
Ubuntu Bionic, installed from source, master

I ran a coverage build for the rcl package on master and realized the coverage results are skipping a test included in master, the file service_fixture.cpp, it is not listed on the coverage report:
Coverage report rcl on master

The test included for this file is added on the CMakelists.txt file as:

  rcl_add_custom_executable(service_fixture${target_suffix}
    SRCS rcl/service_fixture.cpp
    LIBRARIES ${PROJECT_NAME}
    AMENT_DEPENDENCIES ${rmw_implementation} "osrf_testing_tools_cpp" "test_msgs"
  )

  rcl_add_custom_executable(client_fixture${target_suffix}
    SRCS rcl/client_fixture.cpp
    LIBRARIES ${PROJECT_NAME}
    AMENT_DEPENDENCIES ${rmw_implementation} "osrf_testing_tools_cpp" "test_msgs"
  )

  rcl_add_custom_launch_test(test_services
    service_fixture
    client_fixture
    ENV ${rmw_implementation_env_var}
    APPEND_LIBRARY_DIRS ${extra_lib_dirs}
    TIMEOUT 15
  )

Both are tested together, however, it seems only the second one is considered in the coverage result report of CI.

As additional info that could help solving this, generating the coverage results locally using colcon lcov-result shows the file in the coverage report, but it shows the line coverage result as 0%. I think this issue might be related to one of the cmake macros included to add the test, either rcl_add_custom_executable or rcl_add_custom_launch_test.

[Blast545]

@j-rivero Could you take a look to this? You may have a better idea to what's causing this issue and help me fix it

[j-rivero]

@j-rivero Could you take a look to this? You may have a better idea to what's causing this issue and help me fix it

During my time working on the coverage reports I modify and played quite a bit with lcov and github actions but did not really touch the coverage jobs in the buildfarm. Giving a look to the console log I can not find the problem so reproduce the build locally (as you did) would be my next step.

As a side note: the fixtures are being installed and part of the user interfaces provided by the package or are they part of the testing only? If the latter, I might consider them as part of the test code and not relevant for the coverate report.

[Blast545]

Those fixtures are part of the testing suite only. To me, the relevant part was trying to figure out why this was happening because potentially could be a bigger issue. A brief look to coverage CI does not show the service_fixture.cpp, wondering why this happens, I am pointing to an issue with pytest configuration because even if it's a c++ it's actually called with a python script.

I think this could be solved with ament_cmake#226

[j-rivero]

Those fixtures are part of the testing suite only. To me, the relevant part was trying to figure out why this was happening because potentially could be a bigger issue. A brief look to coverage CI does not show the service_fixture.cpp, wondering why this happens, I am pointing to an issue with pytest configuration because even if it's a c++ it's actually called with a python script.

Agree. In fact I run into some problems for coverage using pytest (I finally invoke them without colcon) and that PR could have helped. What happen is you run the coverage using that PR locally? Does the coverage report change somehow?

[Blast545]

When I run coverage locally the report shows the file, but shows the service_fixture.cpp file with 0% of coverage. As if it wasn't run with the coverage tools, but indexed to be taken into account.

[Blast545]

CI infrastructure changed and this is no longer a bug to report, closing this.