Full coverage suite with istanbul

[blinkdog]

The source of all the micro-issues in #61 ...

A full (98+%) coverage suite for the statements, branches, functions, and lines of rot.js.

Instructions for generating the coverage report in node/README.md; requires Node.js.

[ondras]

Wow, impressive!

I have no idea how exactly this works, but it looks like you need to provide (duplicate) server-side unit tests to have almost all lines of the code execute?

Those coffeescript unit tests look suspiciously similar to some of those existing jasmine-based browser tests. Is there a way to reduce this duplicity?

I will merge this anyway, of course.

[ondras]

Thanks for this tedious work!

[blinkdog]

I have no idea how exactly this works, but it looks like you need to provide (duplicate) server-side unit tests to have almost all lines of the code execute?

CoffeeScript has a Cakefile, which is similar to a Makefile, except it's just a CoffeeScript source file with a few helper functions.

cake coverage tells the cake executable to kick off the coverage task, which in our Cakefile is defined as:

task 'coverage', 'Perform test coverage analysis', ->
  clean -> compile -> test -> coverage()

Which is really just cute syntatic sugar for this (JavaScript):

task('coverage', 'Perform test coverage analysis', function() {
  return clean(function() {
    return compile(function() {
      return test(function() {
        return coverage();
      });
    });
  });
});

So first we clean: exec 'rm -fR test'

Then we compile: exec "node_modules/.bin/coffee -o test/ -c node/test"

• This turns all of our CoffeeScript tests in node/test into JavaScript tests in test

Then we test: exec 'node_modules/.bin/mocha --colors --recursive'

• This tells mocha to go run the test suite (by default it looks in the test directory)
• Mostly this is for development purposes; it's not strictly necessary for a coverage report.

Finally we coverage: exec 'node_modules/.bin/istanbul cover node_modules/.bin/_mocha -- --recursive'

• This tells istanbul to instrument the code, so we can figure out which lines, branches, functions, etc. get taken/called or not.
• istanbul then uses mocha to run the test suite.
• After running the tests, it collects the data generated from the instrumented code, and generates a report at coverage/lcov-report/index.html.
• The coverage task also opens the report with firefox: firefox --new-tab coverage/lcov-report/index.html

In the report, you'll see every line, every branch, every function that did/did not get executed.

I organized the test suite according to the source tree. That is, node/test/display/displayTest.coffee corresponds to src/display/display.js. This was to make it easier for developers to know where the tests for a specific piece of code live.

The tests are also organized as follows:

• describe "file"
  • describe "Class"
    • describe "Constructor"
    • describe "Method_1"
    • describe "Method_2"

This was so that finding the specific tests for a very specific part of the source would be easy as well.

Between the report itself and studying the code to figure out how to write a test that would cover each line, each branch, each function, etc. that's why I was able to find so many micro-issues.
Unreachable lines, if statements where it's impossible to reach the else case, etc. stick out like sore thumbs as one writes the necessary tests! :-)

Those coffeescript unit tests look suspiciously similar to some of those existing jasmine-based browser tests. Is there a way to reduce this duplicity?

Yes, mocha suites are very similar to jasmine suites:

• http://mochajs.org/
• https://jasmine.github.io/2.3/introduction.html

The biggest differences (that I've noticed), are that mocha doesn't give you expect, but rather makes you choose your own assertion library, and it doesn't have the nice spy/mocking that jasmine does.

I'd have to study it a little more, but there's probably a way to get the mocha tests to run under the jasmine spec runner. We'd have to figure out where everything should live (and make some nice targets for the Makefile), but we could reduce some of the duplicate tests that way.

I need a short break from writing tests for awhile, but I'll start looking at ways to combine/improve the testing, maybe around the start of June.

All in all, I think it was worth it! :-)

[ondras]

Thanks for this complex explanation. One really learns new stuff every day.