Code Coverage

Code Coverage

The following is a brief explanation of the changes that were made in order to enable a Code Coverage report to be generated and then uploaded to CodeCov after the successful running of tests.

Changes made to CMakeLists.txt

The changes made to CMakeLists.txt to enable code coverage to run are as follows.

option(SOUFFLE_CODE_COVERAGE "Enable coverage reporting" OFF)
...
if(SOUFFLE_CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
    # Code Coverage Configuration
    add_library(coverage_config INTERFACE)
    # Add required flags (GCC & LLVM/Clang)
    target_compile_options(coverage_config INTERFACE
        -O0        # no optimization
        -g         # generate debug info
        --coverage # sets all required flags
    )
    if(CMAKE_VERSION VERSION_GREATER_EQUAL 3.13)
        target_link_options(coverage_config INTERFACE --coverage)
    else()
        target_link_libraries(coverage_config INTERFACE --coverage)
    endif()
endif(SOUFFLE_CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")

The purpose of this modification is to ensure that when cmake is called with the SOUFFLE_CODE_COVERAGE argument turned ON, we create an interface library called coverage_config that specifies the following compile flags, which are necessary for code coverage to run.

-O0        # no optimization
-g         # generate debug info
--coverage # sets all required flags

As we can see in the following line, by default, the SOUFFLE_CODE_COVERAGE variable is set to OFF.

option(SOUFFLE_CODE_COVERAGE "Enable coverage reporting" OFF)
...

Changes made to src/CmakeLists.txt

The changes made to src/CMakeLists.txt are as follows:

...
if(SOUFFLE_CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
    target_link_libraries(libsouffle PUBLIC coverage_config)
endif(SOUFFLE_CODE_COVERAGE AND CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
...

The above ensures that when the SOUFFLE_CODE_COVERAGE argument is turned ON, we compile the souffle project with the compile arguments specified by the code_coverage interface library (specified in the earlier CMakeLists.txt file).

Changes made to CI-Tests.yml

The changes made to the .github/workflows/CI-Tests.yml GitHub workflow file as follows:

...

Code-Coverage:
  needs: [Code-Style, Ubuntu-CMake, OSX-CMake, Memory-Check]
  timeout-minutes: 150

  runs-on: ubuntu-latest

  steps:
  - name: checkout
    uses: actions/checkout@v2
    with:
      fetch-depth: 0

  - name: install-lcov
    run: sudo apt-get update && sudo apt-get install lcov

  - name: install-deps
    run: sudo sh/setup/install_ubuntu_deps.sh

  - name: setup-with-code-coverage-on
    run: cmake -DSOUFFLE_CODE_COVERAGE=ON -S . -B ./build

  - name: make
    run: cmake --build build -j6

  - name: check
    run: cd build && ctest --output-on-failure --progress -j6

  - name: create-coverage-report
    run: lcov --capture --directory . --output-file coverage.info

  - name: remove-system-files-from-coverage-report
    run: lcov --remove coverage.info '/usr/*' --output-file coverage.info

  - name: upload-coverage-report
    uses: codecov/codecov-action@v2
    with:
      token: ${{ secrets.CODECOV_TOKEN }}
      files: coverage.info

As you can see we've added a new Code-Coverage job that runs after all other jobs in CI-Tests.yml have run successfully. As a result, if any of the following jobs/tests fail, our Code-Coverage job doesn't run.

needs: [Code-Style, Ubuntu-CMake, OSX-CMake, Memory-Check]

Our new GitHub job first fetches our project:

- name: checkout
  uses: actions/checkout@v2
  with:
    fetch-depth: 0

We then install lcov, which we will use later to generate our code coverage report:

- name: install-lcov
  run: sudo apt-get update && sudo apt-get install lcov

We then install the Ubuntu dependencies required to build souffle:

- name: install-deps
  run: sudo sh/setup/install_ubuntu_deps.sh

We then run cmake to set up souffle with code coverage compile flags by setting the SOUFFLE_CODE_COVERAGE variable to ON:

- name: setup-with-code-coverage-on
  run: cmake -DSOUFFLE_CODE_COVERAGE=ON -S . -B ./build

We then build souffle:

- name: make
  run: cmake --build build -j6

We then run our tests in souffle:

- name: check
  run: cd build && ctest --output-on-failure --progress -j6

We then generate our code coverage report, which will be output to coverage.info:

- name: create-coverage-report
  run: lcov --capture --directory . --output-file coverage.info

We then remove system files from our coverage.info report:

- name: remove-system-files-from-coverage-report
  run: lcov --remove coverage.info '/usr/*' --output-file coverage.info

We then upload our coverage report using the CodeCov Github Action. Note that we are using v2 of the GitHub Action, as v1 will be deprecated on 1st Feb 2022.

- name: upload-coverage-report
  uses: codecov/codecov-action@v2
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    files: coverage.info

Also note that we must also specify the CODECOV_TOKEN as a secret in GitHub first in order for the uploading process to work.