Sometimes it's convenient to skip specific tests when running coverage. You can do this by
tagging your test descriptions and setting appropriate filters in the .solcover.js mocha options.
Example
// Mocha test to skip
it("is a gas usage simulation [ @skip-on-coverage ]", async function(){
...
})//.solcover.js
module.exports = {
mocha: {
grep: "@skip-on-coverage", // Find everything with this tag
invert: true // Run the grep's inverse set.
}
}The plugin exposes a set of workflow hooks that let you run arbitrary async logic between the main stages of the coverage generation process. These are useful for tasks like launching secondary services which need to connect to a running ethereum client instance (ex: the Oraclize/Provable bridge), or reading data from the compilation artifacts to run special preparatory steps for your tests.
The stages/hooks are (in order of execution):
| Stage | Post-stage hook |
|---|---|
| After instrumentation, before compile. | onPreCompile |
| After compile contracts | onCompileComplete |
| Launch server | onServerReady |
| Run tests using instrumented artifacts | onTestsComplete |
| Generate istanbul coverage reports | onIstanbulComplete |
The tool's general workflow is:
- Read Solidity contract sources from a standard contracts directory
- Rewrite the sources so the code execution path can be tracked by the opcode monitors.
- Compile the modified sources, without optimization
- Launch a HardhatEVM, attaching special listeners that monitor each opcode execution step
- Run tests to completion.
- Transfer collected data to a coverage reporter & report.
Each hook is passed a config object provided by your plugin's dev platform which will contain
relevant source/artifact paths and network info for that stage.
Example
// .solcover.js
const { provableBridge } = require('./helpers');
async function serverReadyHandler(config){
await provableBridge(config.port);
}
module.exports = {
onServerReady: serverReadyHandler,
}If your project is very large or if you have logic that's gas sensitive, it can be useful to minimize the amount of instrumentation the coverage tool adds to your Solidity code.
Usually you're only interested in line and branch coverage but Istanbul also collects data for individual statements and "functions" (e.g - whether every declared function has been called).
Setting the measureStatementCoverage and/or measureFunctionCoverage options to false can
improve performance, lower the cost of execution and minimize complications that arise from solc's
limits on how large the compilation payload can be.
Some advanced testing strategies benefit from knowing which tests in a suite hit a specific line of code. Examples include:
- mutation testing, where this data lets you select the correct subset of tests to check a mutation with.
- fault localization techniques, where the complete data set is a key input to algorithms that try to guess where bugs might exist in a codebase.
Running the coverage command with --matrix will write a JSON test matrix which maps greppable
test names to each line of code to a file named testMatrix.json in your project's root.
It also generates a mochaOutput.json file which contains test run data similar to that
generated by mocha's built-in JSON reporter.
In combination these data sets can be passed to Joran Honig's tarantula tool which uses a fault localization algorithm to generate 'suspiciousness' ratings for each line of Solidity code in your project.
Coverage does not work with the Hardhat's mocha parallel mode. However, it is possible to parallelize coverage in CI environments that support complex workflows. The core idea is to
- partition the set of test files passed to the coverage task
- split coverage into several concurrent jobs, passing test file targets as arguments using the
--testfilescommand line flag - cache the coverage results in shared storage as each job completes
- combine results in a final step (using the instanbul-combine-updated tool)
There's a nice example of this being done in CircleCI at Synthetix, here.
💡 Pro Tip: Codecov CI will automatically combine coverage reports sent to them as a batch - if you're using that service you don't need to do this yourself.
Istanbul has a command line utility which can be used to set thresholds for different coverage categories and throw an error if your coverage drops below them. (Istanbul is a solidity-coverage dependency so you shouldn't need to install it separately.)
Example
$ npx istanbul check-coverage ./coverage.json --statements 99 --branches 94 --functions 99 --lines 99The temp command line option lets you to specify the name of a disposable folder to
stage the compilation artifacts of instrumented contracts in before the tests are run.
Example
$ hardhat coverage --temp build
By default this folder is called .coverage_artifacts. If you already have
preparatory scripts which run between compilation and the tests, you'll probably
find it inconvenient to modify them to handle an alternate path.
This option allows you to avoid that but it's important to realise that the temp folder is automatically deleted when coverage completes. You shouldn't use it if your preferred build target contains information you want to preserve between test runs.
A custom disposable folder to be used for the contracts can be specified by setting the
coverageContractsTemp
property in the configuration file. If not set, this directory defaults to .coverage_contracts.