Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Support gathering code coverage data for RPC tests with lcov #6813
Bitcoin Core has a nice way to gather code coverage data, though it only runs the unit tests and (optionally) the BitcoinJ tests.
With this update, the RPC tests (via
When cleaning, there were several leftovers, and only the coverage related files in
Because Python is used to run the RPC tests, it is explicitly checked, whether Python is available.
The configuration option
Generating coverage data requires
There are a few configuration options, but no further setup is necessary.
To include the BitcoinJ tests, get the test tool:
The coverage data can then be gathered with:
It runs the tests and generates two HTML reports:
(only line and function coverage, no branch coverage, without extended tests)
It's not yet clear to me where #6804 is going, so this may be complementary. #6804 seems like a very easy and fast way to get an understanding of what's covered by the RPC tests, while this PR is more heavy. Given that the lcov testing already exists, extending it to include RPC tests seems nevertheless reasonable in my opinion.
Agree that this is compatible with #6804; I think this sort of "industrial strength" coverage report is absolutely worth having, but an expedient and simple view into which (if any) RPC functions are going untested is valuable also. Lcov certainly provides a more exact notion of what is tested, but it takes a bit more work to tease out which RPC functions are untested; the simple binary coverage provided in #6804 answers that question immediately and unambiguously, despite being primitive.
#6804 may also make it easier to gate builds down the road based on RPC coverage -- if the list of untested RPC functions calculated is non-empty, we fail. I think this would be nice to do once we have tests in place for each RPC function, It's not immediately apparent to me how we'd do this with just lcov.
Not saying that high test coverage stands for quality. But it is a good indicator for people writing new code that they should write unit tests...
@jamesob: based on what you posted in #6804, the list of uncovered RPCs is pretty large, and I agree, it's nice to have a quick overview. At some point it probably becomes less useful, if a binary outcome doesn't provide further insight (e.g. once all RPCs are rudimentary covered).
@jonasschnelli: this is an interesting idea. Some time ago I actually tried to combine
On a slightly related note: it looks like coveralls is run two times in your repo (Linux+clang, Linux+gcc), and if this is an issue for you, you may look into a custom script deployment, which supports expressive conditionals (I used it to deploy docs).
One question that arrises is about the extra build time, if coverage data were gathered via Travis + Coveralls. I haven't timed it, but I have the impressionen it takes significantly longer to go though the process, so this would be something to consider. I'd like, if it were nicely integrated though.
@dexX7 the console output itself certainly becomes less useful after all RPCs are covered, but I think the lasting value is the ability to easily fail a build if someone introduces a new RPC without writing tests that exercise it as a client.
Until now there were quite a few leftovers, and only the coverage related files in `src/` were cleaned, while the ones in the other dirs remained. `qa/tmp/` is related to the BitcoinJ tests, and `cache/` is related to RPC tests.
Because Python is (going to be) used to run the RPC tests, when gathering coverage data with lcov, it is explicitly checked, whether Python is really available.
When using lcov to gather code coverage data, the configuration option `--enable-extended-rpc-tests` may be used to enable extended RPC tests.
The configuration option `--enable-comparison-tool-reorg-tests` may be used to enable extended tests via BitcoinJ also for coverage testing.
The RPC tests (via `qa/pull-tester/rpc-tests.py`) are now executed, when gathering code coverage data, for example with `make cov`. Generating coverage data requires `lcov`, which can installed with: sudo apt-get install lcov To also use the BitcoinJ tests, get the test tool: TOOL_URL=https://github.com/theuni/bitcoind-comparisontool/raw/master/pull-tests-8c6666f.jar TOOL_HASH=a865332b3827abcde684ab79f5f43c083b0b6a4c97ff5508c79f29fee24f11cd wget $TOOL_URL -O ./share/BitcoindComparisonTool.jar echo "$TOOL_HASH ./share/BitcoindComparisonTool.jar" | shasum --algorithm 256 --check The coverage data can be generated with: ./autogen.sh ./configure --enable-lcov --with-comparison-tool=./share/BitcoindComparisonTool.jar make make cov Optionally the options `--enable-extended-rpc-tests` and `--enable-comparison-tool-reorg-tests` may be used to enable more time consuming tests. It then runs the tests and generates two HTML reports: - test_bitcoin.coverage/index.html - total.coverage/index.html
d80e3cb Support gathering of code coverage data for RPC tests (dexX7) e3b5e6c Run extended BitcoinJ tests for coverage based on config (dexX7) 45d4ff0 Add config option to enable extended RPC tests for code coverage (dexX7) 8e3a27b Require Python for RPC tests, when using lcov (dexX7) d425877 Remove coverage and test related files, when cleaning up (dexX7) 4d2a926 Ignore coverage data related and temporary test files (dexX7)