Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: include doctests in the test coverage of code that tarpaulin calculates #533

Merged
merged 20 commits into from Jun 29, 2023

Conversation

dlaehnemann
Copy link
Member

Using a syntax found here:
xd009642/tarpaulin#538 (comment)

We are also using this in rust-htslib, now.

@coveralls
Copy link

coveralls commented Jun 20, 2023

Coverage Status

coverage: 85.218% (+4.3%) from 80.932% when pulling 889e87d on include-doctests-in-code-coverage-calculation into e2b2fd8 on master.

…b.com:rust-bio/rust-bio into include-doctests-in-code-coverage-calculation
…with rust-htslib

these are probably not needed...
Copy link
Member

@tedil tedil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks quite reasonable to me ;)
I guess while there is a way to somehow mock stdin for doctests, it is not a big issue for now. We could of course manually replace stdin in the examples with some fixed input in the future.

@dlaehnemann
Copy link
Member Author

I guess as an example, the stdin-construct is useful as well. So I'm also not overly concerned about this.

@dlaehnemann dlaehnemann merged commit 29cf0f5 into master Jun 29, 2023
11 checks passed
@dlaehnemann dlaehnemann deleted the include-doctests-in-code-coverage-calculation branch June 29, 2023 08:30
@dcroote
Copy link
Contributor

dcroote commented Jun 29, 2023

What a heroic effort @dlaehnemann !

@dlaehnemann
Copy link
Member Author

Thanks for the kind words! 😅

It did take me a while to figure out that tarpaulin + actions-rs was swallowing up the relevant doctest error output and then figuring out how I could run the same setup locally... If anybody comes looking for info on debugging cargo tarpaulin doctest coverage around here:

  1. Install the nightly toolchain with rustup (needed for tarpaulin doctest coverage).
  2. Turn on --debug and --verbose in the cargo tarpaulin command.
  3. Copy-paste the rustdoc command that is now printed out in the CI, to run it locally and get all the output, for example:
Running `/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustdoc --edition=2018 --crate-type lib --crate-name bio --test src/lib.rs --test-run-directory /home/runner/work/rust-bio/rust-bio -L dependency=/home/runner/work/rust-bio/rust-bio/target/debug/deps -L dependency=/home/runner/work/rust-bio/rust-bio/target/debug/deps --test-args --test-threads --test-args 1 --extern anyhow=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libanyhow-c6cf0643f3c509bb.rlib --extern approx=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libapprox-bfa4e03988db6ef8.rlib --extern bio=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libbio-f8a8caca8f493098.rlib --extern bio_types=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libbio_types-5834d92d1c46e184.rlib --extern bit_set=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libbit_set-26380277f767511f.rlib --extern bv=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libbv-502cf4d8e73630ab.rlib --extern bytecount=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libbytecount-9b439d721f18ee63.rlib --extern csv=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libcsv-3d38631523f1901f.rlib --extern custom_derive=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libcustom_derive-5ff07f976469020f.rlib --extern editdistancek=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libeditdistancek-8c6ff1406d9ca3f4.rlib --extern enum_map=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libenum_map-643d7108588eefc6.rlib --extern fxhash=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libfxhash-b77bbadc3c8ee64c.rlib --extern getset=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libgetset-5d904b9ae15addf1.so --extern itertools=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libitertools-094a968098f292d0.rlib --extern itertools_num=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libitertools_num-995199b6bce6f447.rlib --extern lazy_static=/home/runner/work/rust-bio/rust-bio/target/debug/deps/liblazy_static-d0c2dc85fa90a848.rlib --extern multimap=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libmultimap-5cc0df6d1f5eafcc.rlib --extern ndarray=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libndarray-6c31cbb51e19675f.rlib --extern newtype_derive=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libnewtype_derive-61bcd3628007834a.rlib --extern num_integer=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libnum_integer-cabbc0ef90f1a2e0.rlib --extern num_traits=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libnum_traits-4ed70ee29490e97a.rlib --extern ordered_float=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libordered_float-b407e7b6f8375d07.rlib --extern pest=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libpest-99ec8909eeb71d69.rlib --extern pest_derive=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libpest_derive-e1db1b01905d5ad5.so --extern petgraph=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libpetgraph-21f1893f777ff568.rlib --extern proptest=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libproptest-ae59404c71bd4d08.rlib --extern rand=/home/runner/work/rust-bio/rust-bio/target/debug/deps/librand-f82595de47ce7eac.rlib --extern regex=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libregex-535503b0639f3019.rlib --extern serde=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libserde-f6b47fff014ddb92.rlib --extern serde_derive=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libserde_derive-8449af3a227a832e.so --extern statrs=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libstatrs-635688025de7198c.rlib --extern strum=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libstrum-96436eb227d2763f.rlib --extern strum_macros=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libstrum_macros-9310f4a2b996ea58.so --extern tempfile=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libtempfile-fad2efdb632262d8.rlib --extern thiserror=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libthiserror-e0d424f42cc5b579.rlib --extern triple_accel=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libtriple_accel-d8e30d009751d8d9.rlib --extern vec_map=/home/runner/work/rust-bio/rust-bio/target/debug/deps/libvec_map-f06fa9452ca8d0f8.rlib -C embed-bitcode=no --cfg 'feature="generic-simd"' --cfg 'feature="pest"' --cfg 'feature="pest_derive"' --cfg 'feature="phylogeny"' --cfg 'feature="runtime-dispatch-simd"' -Cdebuginfo=2 --cfg=tarpaulin --persist-doctests /home/runner/work/rust-bio/rust-bio/target/doctests -Zunstable-options -Clink-dead-code --error-format json`

I condensed this down to something like (I think -- I didn't record it while doing it, so this is from memory):

rustdoc --edition=2018 --crate-type lib --crate-name bio --test src/lib.rs --cfg 'feature="generic-simd"' --cfg 'feature="pest"' --cfg 'feature="pest_derive"' --cfg 'feature="phylogeny"' --cfg 'feature="runtime-dispatch-simd"' -Cdebuginfo=2 --cfg=tarpaulin --persist-doctests /home/runner/work/rust-bio/rust-bio/target/doctests -Zunstable-options -Clink-dead-code --error-format human

Also, the setting of --error-format human helps... ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants