Lab 06: Code Coverage

The first example presented here is borrowed from here

So far in this class you have learned how to do Unit Tests. You also learned about Code Coverage and that every line of code should be tested at some point during development. However, keeping manual track of what code has been tested or not is hard. Thankfully, there are many tools that you can use to report which lines of code are covered by tests and which ones aren't. You can even incorporate some of these tools into your CI solution (like Github Actions), but in this case we are going to rely in the one tool that has been there for you since the beginning but you hardly think about: the compiler. That's right, we are going to learn how to use the gcc compiler as a code coverage reporting tool and incorporate it with CMake, so let's get started.

Introducing Gcov

Don't do these steps in the repository but set an offline project of your own. Each team member should follow the next steps to learn how to use Gcov

Gcov is a source code coverage analysis and statement-by-statement profiling tool. Gcov generates exact counts of the number of times each statement in a program is executed and annotates source code to add instrumentation. It is already part of the gcc compiler so it is already available if you have the compiler installed. Let us go ahead and generate our first report. Go ahead and create a file named cov.cpp and add the following to it:

#include <stdio.h>

int main ()
{
  int i;

  for (i = 1; i < 10; i++)
    {
      if (i % 3 == 0)
        printf ("%d is divisible by 3\n", i);
      if (i % 11 == 0)
        printf ("%d is divisible by 11\n", i);
    }

  return 0;
}

After creating the file, we are going to compile it, but we need to enable specific flags for this to work. Run the following on the terminal:

gcc -Wall -fprofile-arcs -ftest-coverage cov.cpp -o coverage_example

This is what each flag means:

• -Wall you should know. This enables all compiler warnings, which as good coders we should always do.
• -fprofile-arcs is the flag that enables the recording of which lines of codes are executed and how many times.
• -ftest-coverage generates a report that can be read by Gcov.

The proper term for the executable generated by this command is instrumented executable. It means that the executable has specific information that can be used for profiling. You should know about special executables since you generated one for your lab on debugging. You probably notice a bunch of files were created, these contain all the data needed for the analysis. Now, you need to run the executable once. Then, the next step is to use gcov to generate the anottated report. So run the following:

$ gcov cov.cpp
 88.89% of 9 source lines executed in file cov.cpp
Creating cov.cpp.gcov

The actual percentage might be a little different on your end. You may go ahead and open the .gcov file generated. It should look something like this:

        -:    0:Source:cov.cpp
        -:    0:Graph:cov.gcno
        -:    0:Data:cov.gcda
        -:    0:Runs:1
        -:    0:Programs:1
        -:    1:#include <stdio.h>
        -:    2:
        1:    3:int main ()
        -:    4:{
        -:    5:  int i;
        -:    6:
       10:    7:  for (i = 1; i < 10; i++)
        -:    8:    {
        9:    9:      if (i % 3 == 0)
        3:   10:        printf ("%d is divisible by 3\n", i);
        9:   11:      if (i % 11 == 0)
    #####:   12:        printf ("%d is divisible by 11\n", i);
        -:   13:    }
        -:   14:
        1:   15:  return 0;
        -:   16:}
        

According to this report, line 12 was never run. Line 10 was run 10 times. Now, how is this reporting which lines of code are covered by unit tests? The answer is it doesn't, at least not yet. So far you have generated a report on which lines of code are executed after a single run. Next, we are going to use gcov to figure out how well our tests are covering the code. Before we go ahead, make sure to keep these files ready to show during your demo. You do not need to commit or push them to the repository but do not delete them so you can still get full credit.

Gcov with CMAKE

You may go ahead and clone this repository. The first thing you will notice is that just like Lab 04, we have multiple folders to keep our files organized and a CMAKE file that helps with compilation. Go ahead and open the CMakeLists.txt.in file. It should look like this:

cmake_minimum_required(VERSION 3.15)

project(googletest-download NONE)

include(ExternalProject)
ExternalProject_Add(googletest
  GIT_REPOSITORY    https://github.com/google/googletest.git
  GIT_TAG           main
  SOURCE_DIR        "${CMAKE_CURRENT_BINARY_DIR}/googletest-src"
  BINARY_DIR        "${CMAKE_CURRENT_BINARY_DIR}/googletest-build"
  CONFIGURE_COMMAND ""
  BUILD_COMMAND     ""
  INSTALL_COMMAND   ""
  TEST_COMMAND      ""
)

What this file is doing is adding information on an external repository that we will use for our project. This repository is for Google Test. This time we are asking CMAKE to add the GTest libraries for us, so no need to add it as a submodule. This is one of the many ways we can add GTest to our project.

Now go ahead and run:

cmake .

Feel free to explore CMakeLists.txt to see how we make it possible for all our files to be linked. At this point you can ignore main.cpp. That code only provides sample usage of the testsubject class.

Inside the test folder you can find tester.cpp and triangletest.cpp. The second one is testing the same class declared in Triangle.h that you used during lab 04. To generate the executables, from your project folder run the make command:

$ make

This will create two executables: main and test. Now run test by typing ./bin/test. We need to run the test EXACTLY ONCE to generate the report.

Now the question is, how do we run Gcov?. The linked files that you need for Gcov is stored inside the CMakeFiles folder. So, run the following commands:

$ cd CMakeFiles/test.dir/src
$ gcov testsubject.cpp.gcno
File '[PROJECT_DIR]/src/testsubject.cpp'
Lines executed:100.00% of 6
$ gcov Triangle.cpp.gcno
File '[PROJECT_DIR]/src/Triangle.cpp'
Lines executed:29.17% of 24
Creating 'Triangle.cpp.gcov'

Notice that we are not generating the report for the test files. We don't care about coverage of test cases but how well the test cases are covering production code. If you open the gcov files you will be able to see which lines of code are covered.

Lcov

While Gcov is effective, when working with several reports it can become cumbersome to have to open each report separately. A solution for this is to use Lcov, which is a tool for condensing Gcov reports and export them in HTML format. The next steps will show you how to do that. Form your src folder, run the following commands:

$ lcov --capture --directory . --output-file gtest_coverage.info
$ genhtml gtest_coverage.info --output-directory CODE_COVERAGE

The first command is looking for all gcov reports in the folder and the second generates the HTML export. If you go inside the folder CODE_COVERAGE you should see many files including an index.html. If you are working on the CS100 server you might only be able to see the HTML code for this file. If that is the case, you should commit and push your changes so you can download this folder onto your laptop. Alternatively, you can the install the HTML Preview extension in VSCode and then right click each HTML file in VSCode and choose Open Preview. Sadly with this method the hyperlinks will not work, but you can navigate through each HTML file by right clicking and choosing Open Preview. With either method, open the file named /src/Triangle.cpp.gcov.html. This will show you the line-by-line report for testing coverage of Triangle.cpp, and at this point only one of the functions is being covered.

Please note that every time you run the executable, and then run the gcov, lcov and genhtml commands, the counter for each line will keep accumulating the number of times the line was run. If you run the executable multiple times, the count for all runs will be added together. In order to get an accurate count for the coverage based on a single run, make sure to first reset the counters by running the following command from the same src folder:

$ lcov --zerocounters --directory .

Then, run your test executable ONCE, followed by the usual gcov, lcov and genhtml commands to generate a new report based on that single run.

Submission

At this point you can copy your tests for Triangle.cpp that you created for lab 04. To complete your lab, you need to do the following steps:

• hailstonetest.cppis empty. Modify it to include tests for Hailstone.cpp. You can use the same tests from lab 04.
• CMakeLists.txt is not including the Hailstone files right now. Modify line 43 so it looks like this:

set(SOURCE_FILES src/testsubject.cpp src/Triangle.cpp src/Hailstone.cpp)

Then go to line 46 and change it to this:

set(TEST_FILES test/tester.cpp test/triangletest.cpp test/hailstonetest.cpp)

• You can run the CMAKE commands again to ensure that a report is being generated for Hailstone.cpp.
• Add more tests so that there are at least three tests for each function in the three cpp files within the src folder (i.e. testsubject.cpp, Triangle.cpp, and Hailstone.cpp). In the end, the report should show that each line is executed at least once by the tests and each function is executed at least three times. Also, make sure that the tests for each function are using different GTest assertion types, not variations of the same assertion.
• Generate the gcov and lcov reports and show the TA that you have 100% coverage.
• Commit and push to a branch called Coverage_Branch

There are many tools that are more friendly and convenient than gcov and lcov (but probably not as resoursce efficient). The best implementation for code coverage is to integrated with your CI solution. There were many step involved in this lab to set up the coverage tool but now you know how to do coverage reports locally and at no cost. For your final project, consider exploring other tools as well.