Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Code coverage tooling #448

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Code coverage tooling #448

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b3e4aa5
grcov wrapper around run-tests.sh
svix-dylan May 19, 2022
78e089f
gitignore and results streamlining
svix-dylan May 19, 2022
8e4de04
feedback
svix-dylan May 19, 2022
7d14cb9
how to use code coverage
svix-dylan May 19, 2022
2ce88c8
escape strings
svix-dylan May 19, 2022
e31fe11
simplify location echo
svix-dylan May 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions server/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,6 @@ Cargo.lock
Session.vim

.env

# grcov outputs
*.profraw
6 changes: 6 additions & 0 deletions server/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,3 +136,9 @@ The easiest way to get these tests to pass is to:
Alternatively, if you're only interested in running unit tests, you can just run `cargo test --lib`. These tests don't make any assumptions about the surrounding environment.

To run only a specific test (e.g. only the application tests), you can use the `--test` flag to `cargo test` which supports common Unix glob patterns. For example: `cargo test --test '*app*'`.

# Code Coverage
To get a code coverage report, first make sure you have the following installations:
- `cargo install grcov`
- `rustup component add llvm-tools-preview`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work if default is not set to nightly. Probably just need to say "use nightly" which will also simplify the rest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran both of these install commands on stable successfully

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, they install fine, but I had issues. Though maybe my issues were unrelated.

Then, run `./run-tests-with-grcov.sh`, which is a wrapper around `./run-tests.sh` that includes code coverage evaluation and report generation using `grcov`. When it finishes, it will print the location of the report HTML file to your shell.
19 changes: 19 additions & 0 deletions server/run-tests-with-grcov.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
#!/bin/sh -e
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the -e needed here?


# cleanly run the following commands in their own session
bash -c "
# tell Rust to run with coverage instrumentation
RUSTFLAGS=\"-Cinstrument-coverage\"
# give grcov a profile name template for output files
LLVM_PROFILE_FILE=\"svix-webhooks-%p-%m.profraw\"
# put the compiler in nightly mode
RUSTC_BOOTSTRAP=1
svix-dylan marked this conversation as resolved.
Show resolved Hide resolved

# run tests
./run-tests.sh
" || true
Comment on lines +4 to +14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you doing this in a subscript like this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that these variables are only set in this new session, and don't affect rustc in any subsequent commands a developer might run unrelated to this script.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you do this:

RUSTFLAGS="-Cinstrument-coverage" \
LLVM_PROFILE_FILE="svix-webhooks-%p-%m.profraw" \
RUSTC_BOOTSTRAP=1 \
./run-tests.sh

...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to do anything, it won't affect anything outside of the script.
Also, I just noticed you didn't run export so actually these won't even work at all.

Comment on lines +4 to +14
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
bash -c "
# tell Rust to run with coverage instrumentation
RUSTFLAGS=\"-Cinstrument-coverage\"
# give grcov a profile name template for output files
LLVM_PROFILE_FILE=\"svix-webhooks-%p-%m.profraw\"
# put the compiler in nightly mode
RUSTC_BOOTSTRAP=1
# run tests
./run-tests.sh
" || true
# tell Rust to run with coverage instrumentation
export RUSTFLAGS="-Cinstrument-coverage"
# give grcov a profile name template for output files
export LLVM_PROFILE_FILE="svix-webhooks-%p-%m.profraw"
export TEST_NIGHTLY="+nightly"
# run tests
./run-tests.sh

And then in run-tests:

TEST_COMMAND="cargo ${TEST_NIGHTLY} test --all --all-features --all-targets -- --test-threads=1"

Copy link
Member

@tasn tasn May 20, 2022 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or better yet, just tell people to use nightly (see other comment), and then:

#!/bin/sh -e

# tell Rust to run with coverage instrumentation
export RUSTFLAGS="-Cinstrument-coverage"

# give grcov a profile name template for output files
export DIRNAME="./target/debug/coverage/$(date -u --iso-8601=seconds)"
mkdir -p "$DIRNAME"
export LLVM_PROFILE_FILE="$DIRNAME/svix-webhooks-%p-%m.profraw"

echo "Collecting coverage in $DIRNAME"

# run tests
./run-tests.sh

# generate and open report output
echo "Generating coverage report (may take time)."
~/.cargo/bin/grcov "$DIRNAME" -s svix-server --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o "$DIRNAME"

echo "Coverage HTML file at:\n$(pwd)/$DIRNAME/index.html"


# generate and open report output
grcov . -s . --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o ./target/debug/coverage/

echo "coverage HTML file at $(pwd)/target/debug/coverage/index.html"