the-100percent-code-coverage-fallacy

This is the companion repo for the article The Fallacy of the 100% code coverage.

It showcases a system having 100% code coverage (as you can see from the coverage badge) using low-quality testing through Assertion Free Tests.

The system is an implementation of the FizzBuzz Kata. It was grown using TDD. Once fully implemented, I removed all assertions (see FizzBuzzTest). The system still has 100% code coverage. But when running the mutation tests none of the mutations is killed, resulting in zero useful tests.

Number of Classes	Line Coverage	Mutation Coverage
1	100% 12/12	0% 0/14

Prerequisites

The system requires a JVM 1.8.

(I'm a bit lazy to upgrade to the latest JVM. Pull requests are welcome :D)

Installation on macos:

• Install jenv for managing JVMs.

  $ brew install jenv
  $ jenv doctor
  [OK] No JAVA_HOME set
  [OK] Java binaries in path are jenv shims
  [OK] Jenv is correctly loaded
  jenv enable-plugin export
  # restart your shell

• Install Java8

  brew install --cask temurin8
  jenv add /Library/Java/JavaVirtualMachines/temurin-8.jdk/Contents/Home/

Code Coverage report

The code coverage is measured using JaCoCo.

To obtain a code coverage report:

./gradlew test jacocoTestReport

The coverage report can be found at build/reports/jacoco/test/html/index.html

Mutation Testing report

The mutation testing is done using PIT.

To obtain a mutation testing report:

./gradlew pitest

The build will fail because not 100% of the mutations were killed. In the report you will see that the code coverage dropped to 0%.

The mutation testing report can be found at build/reports/pitest/YYYYMMddhhmm/index.html