Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Support gathering code coverage data for RPC tests with lcov #6813
Bitcoin Core has a nice way to gather code coverage data, though it only runs the unit tests and (optionally) the BitcoinJ tests.
With this update, the RPC tests (via
When cleaning, there were several leftovers, and only the coverage related files in
Because Python is used to run the RPC tests, it is explicitly checked, whether Python is available.
The configuration option
Generating coverage data requires
There are a few configuration options, but no further setup is necessary.
To include the BitcoinJ tests, get the test tool:
The coverage data can then be gathered with:
It runs the tests and generates two HTML reports:
(only line and function coverage, no branch coverage, without extended tests)
referenced this pull request
Oct 13, 2015
It's not yet clear to me where #6804 is going, so this may be complementary. #6804 seems like a very easy and fast way to get an understanding of what's covered by the RPC tests, while this PR is more heavy. Given that the lcov testing already exists, extending it to include RPC tests seems nevertheless reasonable in my opinion.
Agree that this is compatible with #6804; I think this sort of "industrial strength" coverage report is absolutely worth having, but an expedient and simple view into which (if any) RPC functions are going untested is valuable also. Lcov certainly provides a more exact notion of what is tested, but it takes a bit more work to tease out which RPC functions are untested; the simple binary coverage provided in #6804 answers that question immediately and unambiguously, despite being primitive.
#6804 may also make it easier to gate builds down the road based on RPC coverage -- if the list of untested RPC functions calculated is non-empty, we fail. I think this would be nice to do once we have tests in place for each RPC function, It's not immediately apparent to me how we'd do this with just lcov.
Not saying that high test coverage stands for quality. But it is a good indicator for people writing new code that they should write unit tests...
@jamesob: based on what you posted in #6804, the list of uncovered RPCs is pretty large, and I agree, it's nice to have a quick overview. At some point it probably becomes less useful, if a binary outcome doesn't provide further insight (e.g. once all RPCs are rudimentary covered).
@jonasschnelli: this is an interesting idea. Some time ago I actually tried to combine
On a slightly related note: it looks like coveralls is run two times in your repo (Linux+clang, Linux+gcc), and if this is an issue for you, you may look into a custom script deployment, which supports expressive conditionals (I used it to deploy docs).
One question that arrises is about the extra build time, if coverage data were gathered via Travis + Coveralls. I haven't timed it, but I have the impressionen it takes significantly longer to go though the process, so this would be something to consider. I'd like, if it were nicely integrated though.
@dexX7 the console output itself certainly becomes less useful after all RPCs are covered, but I think the lasting value is the ability to easily fail a build if someone introduces a new RPC without writing tests that exercise it as a client.