Find file History

Dextool Mutate

Dextool's plugin for mutation testing of C/C++ projects. It can help you design new tests and evaluate the quality of existing tests by measuring their ability to detect artificially injected faults.


  • Provides support for conventional mutation operators: AOR, ROR, DCC, DCR, LCR, COR, SDL.
  • Can continue from where a testing session was interrupted.
  • Allows multiple instances to be run in parallel.
  • Makes type aware ROR to reduce the number of equivalent mutants.
  • Can reuse previous results when a subset of the SUT changes by only testing those changes (files for now).
  • Provides multiple report formats (Markdown, compiler warnings, JSON).
  • Lets a user modify it by using a SQLite database as intermediary storage.
  • Can rerun e.g. the mutations that previously survived when new tests are added to the test suite.
  • Does automatic handling of infinite loops (timeout).
  • Works with all C++ versions.
  • Works with C++ templates.
  • Has a simple workflow.
  • Integrates without modifications to the projects build system.
  • Detect test cases that do not kill any mutants. These test cases most probably do not verify anything.
  • Detect test cases that kill the same mutants. These is an indication of redundant test cases.

Mutation Testing

This section explains how to use Dextool Mutate to analyze a C++ project that uses the CMake build system.

Note though that the Dextool work with any build system that is able to generate a JSON compilation database. It is just that CMake conveniently has builtin support to generate those. For other build systems there exists the excellent BEAR tool that can spy on the build process to generate such a database.

The Google Test project is used as an example.

Obtain the project you want to analyze:

git clone
cd googletest

Generate a JSON compilation database for the project:

mkdir build
pushd build
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -Dgtest_build_tests=ON -Dgmock_build_tests=ON ..

Create a configuration file:

dextool mutate admin --init

Open the config file and change the following fields:

restrict = ["googlemock/include", "googlemock/src", "googletest/include", "googletest/src"]
extra_flags = [ "-D_POSIX_PATH_MAX=1024" ]
search_paths = ["./build/compile_commands.json"]
test_cmd = ""
build_cmd = ""
analyze_using_builtin = ["gtest", "ctest"]

Generate a database of all mutation points:

dextool mutate analyze

Create a file that will build the subject under test when invoked:

set -e
cd build
make -j$(nproc)

Create a file that will run the entire test suite when invoked:

set -e
cd build
ctest --output-on-failure

Make the files executable so they can be used by dextool:

chmod 755

Run the mutation testing on the LCR mutants:

dextool mutate test --mutant lcr

For more examples see here.

Custom Test Analyzer

Create a file that will identify a failing test from stdout:

# The arguments are paths to stdout ($1) and stderr ($2).
grep -h "(Failed)" $1 $2

Don't forget to make it executable:

chmod 755

And configure dextool to use it. Either via CLI (--test-case-analyze-cmd) or config:

analyze_cmd = ""

Parallel Run

It is possible to run multiple instances of dextool the same database. Just make sure they don't mutate the same source code.


To see the result of the mutation testing and thus specifically those that survived it is recommended to user the preconfigured --level alive parameter. It prints a summary and the mutants that survived.

dextool mutate report --level alive --mutant lcr

But it is possible to in more detail control what sections are printed for the --plain printer. Lets say we want to print the test case statistics, the summary and the killed mutants.

dextool mutate report --section tc_stat --section summary --section killed --section tc_killed_no_mutants --mutant lcr

See --section for a specification of the supported sections.

Re-test Alive Mutants

Lets say that we want to re-test the mutants that survived because new tests have been added to the test suite. To speed up the mutation testing run we don't want to test all mutants but just those that are currently marked as alive.

This can be achieved by resetting the status of the alive mutants to unknown followed by running the mutation testing again.

Example of resetting:

dextool mutate admin --mutant lcr --operation resetMutant --status alive

Incremental Mutation Testing

The tool have support for testing only the changes to a program by reusing a previous database containing mutation testning result. All we have to do to use this feature is to re-analyze the software. The tool will then remove all the mutants for files that have changed.

Code Coverage

It may be interesting to compare mutation testing results with code coverage. To measure code coverage for the Google Test project, build it with:

cmake -DCMAKE_CXX_FLAGS="-fprofile-arcs -ftest-coverage" -DCMAKE_C_FLAGS="-fprofile-arcs -ftest-coverage" -DCMAKE_EXE_LINKER_FLAGS="-fprofile-arcs -ftest-coverage" -Dgtest_build_tests=ON -Dgmock_build_tests=ON ..

To generate a HTML coverage report:

lcov -c --gcov-tool /usr/bin/gcov -d . --output-file
genhtml -o html