Code coverage

[mbr]

Inspired by @DrPeterVanNostrand's potential bug due to a never-taken branch, I gave coverage a try. Other crates are using some form of coverage (e.g. https://github.com/ctz/rustls does not only feature an awesome logo, but over a hundred lines of custom tooling to enable this functionality), but it is far from trivial to get it working.

Here's what I tried this morning, by trying to get coverage for a toy crate that uses some crates and features we want in hbbft in the future:

• cargo-tarpaulin fails with a compiler error when proptest is used.
• cargo-kcov fails with the same compiler error! I filed Encountered error Unimplemented selecting Binder(...) during trans trying to get coverage rust-lang/rust#52478 in response.
• Manually generating output using -Z profile, as described by https://users.rust-lang.org/t/howto-generating-a-branch-coverage-report/8524. Note that the compiler patch is no longer necessary, the post is from 2015 and it seems the necessary patches have been merged into mainline rustc.
  The procedure is quite a bit of work to get going, in the end I did manage to get some output; unfortunately it shows way too much coverage (I got 100% on my first try) and has an issue with macros. As macros are essential (proptest!), I do not expect this to work well in practice.
• cargo-cov fails compilation (filed Compilation fails for master branch, also using cargo install kennytm/cov#22).
• Running kcov manually (https://users.rust-lang.org/t/tutorial-how-to-collect-test-coverages-for-rust-project/650): Segfaults when calling kcov target/cov target/debug/, at least with my kcov (Debian ships an ancient version 11, when 35 is current, so I had to compile by hand).

There are more things to try (e.g. doing a clean step-by-step of https://jbp.io/2017/07/19/measuring-test-coverage-of-rust-programs), but as a whole, I am not very happy about the current state of tooling in this regard.

I am doing something wrong though because shortly before me giving up, this finally worked:

1. Install cargo-make (cargo install cargo-make),
2. run cargo test --no-run to build the tests,
3. create coverage info using cargo make coverage-flow.

Now it is possible to open target/coverage/index.html a browser.

Probable next steps:

• Getting cargo-make to work with tests compiled with --release (will take too long otherwise).
• Checking if the whole thing even runs on hbbft.
• Double-checking whether the output is correct.
• Finding a way to conveniently display the results, either by building HTML files (gh_pages or using some service like https://coveralls.io/).

Comments welcome!

[vkomenda]

cargo make coverage-flow only computes coverage of the internal hbbft tests for me:

running 3 tests
test broadcast::merkle::tests::test_merkle ... ok
test dynamic_honey_badger::votes::tests::test_committed_votes ... ok
test dynamic_honey_badger::votes::tests::test_pending_votes ... ok

This is due to the setting of the test coverage filter being set as follows:

CARGO_MAKE_TEST_COVERAGE_BINARY_FILTER_REGEX=hbbft-[a-z0-9]*$\|test_[a-z0-9]*-[a-z0-9]*$\|[a-z0-9]*_test-[a-z0-9]*$

This variable is set in the initial environment of cargo-make. This could be made to work for the integration tests by appending _test to every test file name.

I think that computing coverage on all executable files in target/release would be even better. There shouldn't be any naming constraints on test file names.

[vkomenda]

For some reason cargo make coverage, when applied to all executable files in target/release, returns 0% coverage for all integration tests and 91.2% for the unit tests in the hbbft binary. However all those executables are compiled with debug info and are not stripped. It would be interesting to know why it still works on the main binary.

It does work for the integration tests in target/debug. The execution time is completely unrealistic though.

[vkomenda]

It took two and a half days on my laptop to run all tests in the current master in debug mode to compute coverage with cargo make coverage: 84.5%. Not a bad figure for a start. Let's see how the proptest-based test framework improves on that mark.

cargo make coverage always runs an installation script to set up kcov. This looks unnecessary in our Travis. We could install kcov when setting up the image and then just use kcov without the cargo-make tool.