Coverage.jl

"Take Julia code coverage and memory allocation results, do useful things with them"

Code coverage: Julia can track how many times, if any, each line of your code is run. This is useful for measuring how much of your code base your tests actually test, and can reveal the parts of your code that are not tested and might be hiding a bug. You can use Coverage.jl to summarize the results of this tracking, or to send them to a service like Coveralls.io or Codecov.io.

Memory allocation: Julia can track how much memory is allocated by each line of your code. This can reveal problems like type instability, or operations that you might have thought were cheap (in terms of memory allocated) but aren't (i.e. accidental copying).

Comparison of coverage packages

• Coverage.jl (this package): allows you to take coverage results and submit them to online web services such as Codecov.io and Coveralls.io
• CoverageTools.jl: core functionality for processing code coverage and memory allocation results

Most users will want to use Coverage.jl.

Working locally

Code coverage

Step 1: collect coverage data. If you are using your default test suite, you can collect coverage data with Pkg.test("MyPkg"; coverage=true). Alternatively, you can collect coverage data manually: in the terminal, navigate to whatever directory you want to start from as the working directory, and run julia with the --code-coverage option:

julia --code-coverage=user

or more comprehensively (if you're interested in getting coverage for Julia's standard libraries)

julia --code-coverage=tracefile-%p.info --code-coverage=user  # available in Julia v1.1+

You can add other options (e.g., --project) as needed. After the REPL starts, execute whatever commands you wish, and then quit Julia. Coverage data are written to files when Julia exits.

Step 2: collect summary statistics (optional). Navigate to the top-level directory of your package, restart Julia (with no special flags) and analyze your code coverage:

using Coverage
# process '*.cov' files
coverage = process_folder() # defaults to src/; alternatively, supply the folder name as argument
coverage = append!(coverage, process_folder("deps"))  # useful if you want to analyze more than just src/
# process '*.info' files, if you collected them
coverage = merge_coverage_counts(coverage, filter!(
    let prefixes = (joinpath(pwd(), "src", ""),
                    joinpath(pwd(), "deps", ""))
        c -> any(p -> startswith(c.filename, p), prefixes)
    end,
    LCOV.readfolder("test")))
# Get total coverage for all Julia files
covered_lines, total_lines = get_summary(coverage)
# Or process a single file
@show get_summary(process_file(joinpath("src", "MyPkg.jl")))

The fraction of total coverage is equal to covered_lines/total_lines.

Step 3: identify uncovered lines (optional). To discover which functions lack testing, browse through the *.cov files in your src/ directory and look for lines starting with - or 0, which mark lines that were never executed. Numbers larger than 0 are counts of the number of times the respective line was executed. Note that blank lines, comments, lines with end statements, etc. are marked with - but do not count against your coverage.

Be aware of a few limitations:

• a line that can take one of two branches gets marked as covered even if only one branch is tested
• currently, code run by Julia's internal interpreter is not marked as covered.

Exclude specific lines or sections from coverage

To exclude specific code blocks, surround the section with COV_EXCL_START and COV_EXCL_STOP comments:

# COV_EXCL_START
foo() = nothing
# COV_EXCL_STOP

To exclude a single line, add a comment with COV_EXCL_LINE:

const a = 1  # COV_EXCL_LINE

Memory allocation

Start julia with

julia --track-allocation=user

Then:

• Run whatever commands you wish to test. This first run is to ensure that everything is compiled (because compilation allocates memory).
• Call Profile.clear_malloc_data()
• Run your commands again
• Quit julia

Finally, navigate to the directory holding your source code. Start julia (without command-line flags), and analyze the results using

using Coverage
analyze_malloc(dirnames)  # could be "." for the current directory, or "src", etc.

This will return a vector of MallocInfo objects, specifying the number of bytes allocated, the file name, and the line number. These are sorted in increasing order of allocation size.

LCOV export

There are many tools to work with LCOV info-format files as generated by the geninfo tool. Coverage.jl can generate these files:

coverage = process_folder()
LCOV.writefile("coverage-lcov.info", coverage)

Cleaning up .cov files

When using Coverage.jl locally, over time a lot of .cov files can accumulate. Coverage.jl provides the clean_folder and clean_file methods to either clean up all .cov files in a directory (and subdirectories) or only clean the .cov files associated with a specific source file.

Tracking Coverage with Codecov.io

Codecov.io is a test coverage tracking tool that integrates with your continuous integration servers (e.g. TravisCI) or with HTTP POSTs from your very own computer at home.

1. Enable Codecov.io for your repository.

   • If it is public on GitHub and you are using using Travis, CircleCI or Appveyor, this is all you need to do. You can sign into Codecov using your Github identity.
   • Otherwise you will need to define a CODECOV_TOKEN environment variable with the Repository Upload Token available under the Codecov settings.

2. Use the command line option when you run your tests:

   • Either with something like julia --code-coverage test/runtests.jl, or
   • with something like julia -e 'Pkg.test("MyPkg", coverage=true)'

3. Configure your CI service to upload coverage data:

   • If you are using Travis with language: julia, simply add codecov: true to your .travis.yml.

   • You can also add the following to the end of your .travis.yml. This line downloads this package, collects the per-file coverage data, then bundles it up and submits to Codecov. Coverage.jl assumes that the working directory is the package directory, so it changes to that first (so don't forget to replace MyPkg with your package's name!

   • On Travis CI:

     after_success:
     - julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Codecov.submit(process_folder())'

   • On AppVeyor:

     after_test:
     - C:\projects\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Codecov.submit(process_folder())"

   • If you're running coverage on your own machine and want to upload results to Codecov, make a bash script like the following:

     #!/bin/bash
     CODECOV_TOKEN=$YOUR_TOKEN_HERE julia -e 'using Pkg; using Coverage; Codecov.submit_local(process_folder())'

Tracking Coverage with Coveralls.io

Coveralls.io is a test coverage tracking tool that integrates with your continuous integration solution (e.g. TravisCI).

1. Enable Coveralls.io for your repository. If it is public on GitHub and you are using TravisCI, this is all you need to do. If you are using AppVeyor, you need to add a secure environment variable called COVERALLS_TOKEN to your .appveyor.yml (see here). Your repo token can be found in your Coveralls repo settings. If neither of these are true, please submit an issue, and we can work on adding additional functionality for your use case.

2. Activate the --code-coverage command line option when you run your tests

   • Either with something like julia --code-coverage test/runtests.jl, or
   • with something like julia -e 'Pkg.test("MyPkg", coverage=true)'

3. Configure your CI service to upload coverage data:

   • If you are using Travis with language: julia, simply add coveralls: true to your .travis.yml.

   • You can also add the following to the end of your .travis.yml. This line downloads this package, collects the per-file coverage data, then bundles it up and submits to Coveralls. Coverage.jl assumes that the working directory is the package directory, so it changes to that first (so don't forget to replace MyPkg with your package's name!

   • On Travis CI:

     after_success:
     - julia -e 'using Pkg; Pkg.add("Coverage"); using Coverage; Coveralls.submit(process_folder())'

   • On AppVeyor:

     after_test:
     - C:\projects\julia\bin\julia -e "using Pkg; Pkg.add(\"Coverage\"); using Coverage; Coveralls.submit(process_folder())"

A note for advanced users

Coverage tracking in Julia is not yet quite perfect. One problem is that (at least in certain scenarios), the coverage data emitted by Julia does not mark functions which are never called (and thus are not covered) as code. Thus, they end up being treated like comments, and are not counted as uncovered code, even though they clearly are. This can arbitrarily inflate coverage scores, and in the extreme case could even result in a project showing 100% coverage even though it contains not a single test.

To overcome this, Coverage.jl applies a workaround which ensures that all lines of code in all functions of your project are properly marked as "this is code". This resolves the problem of over reporting coverage.

Unfortunately, this workaround itself can have negative consequences, and lead to under reporting coverage, for the following reason: when Julia compiles code with inlining and optimizations, it can happen that some lines of Julia code do not correspond to any generated machine code; in that case, Julia's code coverage tracking will never mark these lines as executed, and also won't mark them as code. One may now argue whether this is a bug in itself or not, but that's how it is, and normally would be fine -- except that our workaround now does mark these lines as code, but code which now never has a chance as being marked as executed.

We may be able to improve our workaround to deal with this better in the future (see also #188), but this has not yet been done and it is unclear whether it will take care of all instances. Even better would be if Julia improved the coverage information it produces to be on par with what e.g. C compilers like GCC and clang produce. Since it is unclear when or if any of these will happen, we have added an expert option which allows Julia module owners to disable our workaround code, by setting the environment variable DISABLE_AMEND_COVERAGE_FROM_SRC to yes.

For Travis, this can be achieved by adding the following to .travis.yml:

env:
  global:
    - DISABLE_AMEND_COVERAGE_FROM_SRC=yes

For AppVeyor, add this to .appveyor.yml:

environment:
  DISABLE_AMEND_COVERAGE_FROM_SRC: yes

Some Julia packages using Coverage.jl

Pull requests to add your package welcome (or open an issue)

• ArgParse.jl
• AstroLib.jl
• AudioIO.jl
• Augur.jl
• Bootstrap.jl
• CAIRS.jl
• ClimateTools.jl
• DASSL.jl
• DataFrames.jl
• Decimals.jl
• Distributions.jl
• DSP.jl
• ExtractMacro.jl
• FastaIO.jl
• FiniteStateMachine.jl
• FourierFlows.jl
• Gadfly.jl
• GeometricalPredicates.jl
• Glob.jl
• GradientBoost.jl
• GraphCentrality.jl
• GraphLayout.jl
• Homebrew.jl
• HttpParser.jl
• IntervalTrees.jl
• IPNets.jl
• JointMoments.jl
• JuMP.jl
• LibGit2.jl
• LightGraphs.jl
• LinearExpressions.jl
• Orchestra.jl
• ODE.jl
• OnlineStats.jl
• OpenCL.jl
• OpenStreetMap.jl
• PValueAdjust.jl
• QuantEcon.jl
• QuantileRegression.jl
• RationalSimplex.jl
• RDF.jl
• Requests.jl
• Restful.jl
• Robotics.jl
• RouletteWheels.jl
• SASLib.jl
• SimJulia.jl
• SIUnits.jl
• StatsBase.jl
• TaylorIntegration.jl
• TaylorSeries.jl
• TextWrap.jl
• TimeData.jl
• TypeCheck.jl
• Unitful.jl
• URIParser.jl
• URITemplate.jl
• Voting.jl
• WAV.jl
• Weave.jl
• WeightedStats.jl
• YAML.jl