Microcoverage

This module computes code coverage for a Julia program at a more fine-grained level than the built-in coverage feature. Specifically, it provides coverage counts for each branch of the ||, && and ?: operators where they occur. It also counts the number of invocations to statement-functions.

Usage

Install the software in some directory and then include it at the REPL level (outermost level, either in the interpreter or in a file included by the interpreter not inside a module):

include("microcoverage.jl")

In order to refer to the functions in the module without prefixing them with the module name, use the following declaration:

using microcoverage

Next, instruct the module to instrument your source code:

begintrack("mysourcecode.jl")
begintrack("myothersourcecode.jl")

Now run your code as you normally would. Suppose, for example, that including mytestruns.jl invokes many routines whose source code sits inside mysourcecode.jl and myothersourcecode.jl, so you generate these invocations:

include("mytestruns.jl")

Finally, generate the reports:

endtrack("mysourcecode.jl")
endtrack("myothersourcecode.jl")

Under the hood

The microcoverage module works at the source-code level (as opposed to the standard library coverage feature, which operates close to the machine). The begintrack function copies your source code to a backup file (in the above example, the backup files would be called mysourcecode.jl.orig and myothersourcecode.jl.orig). Then it generates a new source code file (in the above case, named mysourcecode.jl and myothersourcecode.jl) that is peppered with calls to a routine to increment counters. The method used to generate the new source file is as follows. First, the entire file is passed through the parse function. Then the expressions generated by the parse function are fathomed by a routine that inserts a call to increment a counter each time a new source line is encountered and each time one of the aforementioned operators is encountered.

This rewritten source code consists of opaque eval statements and is not meant to be human-readable. The endtrack function restores your original file and generates the report, which shows the source code line and the corresponding counter. The reports have the extension mcov appended; in the above example, the reports would be named mysourcecode.jl.mcov and myothersourcecode.jl.mcov.

Interpreting the report

Here are some examples of lines from the .mcov file and what they mean:

                * function cmp3(o::Ordering,
                *               treenode::TreeNode,
                *               k,
                *               isleaf::Bool)
L167      70360  ? ( 1640 ) : ( 68720 ( 68720 ( 68720 ) && ( 34623 )) || ( 68688 ) ? ( 716 ) : ( 68004 ))
                *     (lt(o, k, treenode.splitkey1))? 1 :
                *     (((isleaf && treenode.child3 == 2) ||
                *       lt(o, k, treenode.splitkey2))? 2 : 3)
                * end

All of these lines are copies of source lines (source lines are preceded with an asterisk) except for the line marked L167. This line is interpreted as follows: L167 means source line number 167. The line was executed 70360 times. The line has a ?: operator. The first branch of the operator was executed 1640 times while the second was executed 68720 times. Meanwhile, the second branch involves an || operator; the first argument of this || operator was executed 68720 times while the second was executed 68688 times. These branches have further nested calling inside of them.

For statement functions, the coverage routine tells how many times they were invoked:

L195      10 ( 10 ) && ( 6 )(called 10 time(s))
                * eq(o::Ordering, a, b) = !lt(o, a, b) && !lt(o, b, a)

This statement function was invoked 10 times. It has an internal branch; the first branch was invoked 10 times, while the second was invoked 6 times.

Limitations

• The microcoverage module uses several undocumented aspects of the Expr type and the parse function. These aspects were discovered via trial and error. This means that they may change in a future version of Julia, so the module is rather fragile.

• The module must be loaded at the REPL level, not inside any other module. The reason is that the invocations to the counter-incrementing routine that are scattered through the instrumented code are of the following form: Main.microcoverage.incrtrackarray(nn). Therefore, if the microcoverage module is nested inside of some other module, then the incrtrackarray function won't be found.

• The package does not work if the instrumented code is run in a forked process. This is because the global variable associated with the incrtrackarray routine will not be known to the other process. In particular, this means that the microcoverage module does not work if the instrumented code is run via Julia's package-testing mechanism: Pkg.test("mymodule"). Instead, it is necessary to run the test within the same process using a statement like this:

  include(joinpath(Pkg.dir("mymodule"), "test", "runtests.jl"))
  

• Once a begintrack instruction is executed, the microcoverage module should not be reloaded until after the corresponding call to endtrack because the global variables keeping track of the instrumented code are lost during the reloading process. If it is necessary to reload microcoverage after a begintrack instruction, then the source code should be restored using the restore function provided in the module, as in the following snippet:

  include("microcoverage.jl")
  using microcoverage
  begintrack("mysourcecode.jl")
  include("microcoverage.jl")   # oops, global variables reset
                                # knowledge of mysourcecode lost!
  using microcoverage
  restore("mysourcecode.jl")    # restore the original version
  begintrack("mysourcecode.jl") # should be good to go now