Skip to content

Test Coverage

Marco Terzer edited this page Dec 5, 2016 · 56 revisions

Test Coverage Overview

decimal4j test coverage overview

Testing Strategy

We know that test coverage does not guarantee bug free code. But we do believe that lack of coverage is almost a guarantee for bugs!

Hence we have tried hard to achieve a high test coverage for the decimal4j library. We are testing all arithmetic operations, all scales, all rounding modes etc. Most of our tests operate on two test sets: random tests and special case tests.

The random tests simply generate random input values to test a certain function. We usually assert our actual results against the expected value derived from an equivalent operation of the BigDecimal (the comprehensiveness of our testing has lead to finding and reporting JDK bugs in the BigDecimal class!)

The special case tests are constructed from special input values such as zero, one, minus one, powers of 10, integer and long min/max values etc. We also randomly choose mutable and immutable Decimal values and alter between equivalent alternatives to perform the tested operation.

All of this does not come for free: a full coverage build takes 13 hours on our build server! This is why we have introduced different test variants for decimal4j builds. For more information on test variants see next section.

Testing Variants

Test variants are controlled via system properties when running the build, e.g.

gradle clean jacoco test -DtestVariant=SMALL
Feature / Variant TINY SMALL STANDARD LARGE ALL
Build Duration <2min ~3min ~5min 20-30min 12-13h
Tested Scales 0,9,18 0,6,9,18 0,6,9,18 0,6,9,17,18 0-18
Rounding Modes DOWN, HALF_UP DOWN, HALF_UP DOWN, HALF_UP, HALF_EVEN UP, DOWN, HALF_UP, HALF_EVEN, UNNEC. (all)
Rounding Modes + Overflow Check DOWN, UNNEC. DOWN, HALF_UP, UNNEC. DOWN, HALF_UP, UNNEC. UP, DOWN, HALF_UP, HALF_EVEN, UNNEC. (all)
Random Tests* 100 1000 1000 10000 10000
Special Values** 43 67 99 171 231

* Random test count per unit test

** Note that the number of special cases has a big impact on the build performance because the number is squared for operations with two arguments!

Test scales can be controlled separately, to run STANDARD test with ALL scales:

gradle clean jacoco test -DtestScales=ALL

Test cases can also be chosen separately, which impacts the special cases and # of random tests:

gradle clean jacoco test -DtestCases=ALL

Rounding modes and overflow modes are determined by the test variant only.

You can’t perform that action at this time.