Fetching contributors…
Cannot retrieve contributors at this time
422 lines (226 sloc) 6.42 KB

Code Coverage and Testing

Dr. Gleb Bahmutov, PhD

uTest -> Applause @bahmutov

Tools at

UTest -> Applause

In the wild testing for mobile / web apps

  • real people
  • real devices
  • real locations

Important to have base quality level before handing off an app to the testers.

Axiom 1

If collecting code coverage is hard - it will be skipped

Axiom 2 - tests add weight

Tractor pull

Test classification

How Google Tests Software

Code Coverage usefulness

  • small tests - extremely useful
  • medium tests - less helpful
  • large tests - not helpful

Small tests

function add(a, b) {
  return a + b;

function sub(a, b) {
  return a - b;

test('addition', function () {
  equal(add(2, 3), 5, '2 + 3 = 5');
  equal(add(-2, 2), 0, '-2 + 2 = 0');

Code vs small tests

Code and tests

Coverage tools

JavaScript and related istanbul

Java and related Emma

Everything else: see


gt math.js
gt tests/**/s*.js
gt -t "add*" math.js
gt -m "math" lots-of-tests.js

coverage is collected by default in all cases

Code coverage use 1

Detect features not covered by tests


coverage 1

Added subtraction test coverage 2

Don't forget branches abs coverage

My personal target

  • 85% for the code

  • 95% for the tests

  • Use Jenkins to check the coverage level
  • Keep tests DRY

Jenkins Cobertura plugin

Jenkins coverage

Small tests coverage challenges

  • avoid coverage by accident
  • covering all paths
    • IE7 + print media CSS + small screen in portrait

Developers are stateless

sub as add

Test skipped but code is still covered


Covering all paths

// individual coverage info file
DA:1,1  //-- line number 1, covered
DA:6,0  //-- line number 6, not covered

Combine results

If tests run in multiple environments, there should be a way to combine results to show unified results.

coverage-ie OR coverage-chrome = coverage-all

Code coverage use 2

Show technical debt

Code Complexity

  function add(a, b) { return a + b }
  // cyclomatic = 1
  // halstead volume = 2

  function abs(a) {
    return a >= 0 ? a : -a
  // cyclomatic = 2
  // halstead volume = 3

jsc, grunt-complexity, complexity-report, plato

risk-map = coverage + complexity


Code coverage use 3

Speed up test execution

Test overlap analysis

  1. Run each unit test separately
  2. Keep track of source files covered by each unit test

sub as add

Test / code overlap by file

                   add     sub
addition test      100%    0%
subtraction test   100%    100%

If running subtraction test, do not run addition test

At least, run subtraction test first

Test / code overlap among files

Every test updates local untested database

$ gt mathTests.js
// mathTests.js covers math.js AND utils.js

$ gt utilsTest.js
// utilsTest.js covers utils.js

// edit utils.js
$ untested
// runs utilsTest.js AND mathTests.js

works great with pre-git hooks


  • Keeping tests up to date
  • Code coverage collection
    • Speed, flexibility
    • Multi-environment setup
  • Fundamental limit
  • Code coverage vs input coverage

Code base size and speed

    10k source files
    2k source files
    1 changed file
  10k source files

Avoid long preprocess step!

Fundamental problem

100% code coverage means nothing and is very hard to get


var foo = function () {
  foo = null;
  return true;

test('foo', funciton () {
  ok(foo(), 'foo returns true');
  // hmm, what if we call foo again?

Meaningless 100%

Checking email format

var emailRegex =


Official RFC 2822 standard email regex is over 400 characters long!

Switch to data coverage

test('valid emails', function () {

test('invalid emails', function () {

Default options using code

Options in code

Default options using object

Options object

Obligatory math

N * 0.8 ⋘ N/2 * 0.8 + N/2 * 0.8

Split a large file and cover at same ratio.

Medium tests

Medium tests

Medium tests

Tests check how pieces come together, some pieces are mocks.

Covering all code paths seems pointless.

Large tests

Entire puzzle

Large tests

Tests try to verify that the user features are working.

Covering all source code is impossible.

note: top level project can still set a test coverage limit for each of its parts.

Cross language / system barriers

Cross language border

Unused and 3rd party code

Small amount of code is used

worse in reality

mix of code

Unit testing can be exhaustive

Full testing

Large testing is limited

Limited testing


  • Set up code coverage to be as easy as pie
  • Use code coverage with small tests
    • combine with complexity
    • speed up tests
    • remember limitations
  • Do not shoot for 100% code coverage

[slides-now-footer]: "@bahmutov code coverage and testing" [slides-now-theme]: "bw" [slides-now-timer]: "40"