Improve test coverage #95
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Gus Monod <gmonod1@bloomberg.net>
This is only needed when building with coverage support, not when running the tests in coverage mode, and the build was performed (with this environment variable set) by an earlier step. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
The test suite includes several dummy programs whose whole purpose is to be attached to in order to inspect their state. It's hard to count coverage for these, because they're run in subprocesses, and because the interpreter used to run that subprocess might not even be the interpreter for which `coverage` is installed. Just exclude those files from coverage reports for now, rather than trying to find some way to force Coverage to trace them. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
godlygeek
force-pushed
the
cover-tests-better
branch
from
2ed442a
to
c89e680
Compare
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #95 +/- ##
==========================================
+ Coverage 82.48% 89.13% +6.65%
==========================================
Files 27 46 +19
Lines 3260 5432 +2172
Branches 226 875 +649
==========================================
+ Hits 2689 4842 +2153
- Misses 571 589 +18
- Partials 0 1 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
☔ View full report in Codecov by Sentry. |
sarahmonod
reviewed
May 3, 2023
sarahmonod
reviewed
May 3, 2023
Some tests have branches to make different assertions depending on the version of the interpreter that is being attached to. Because we only run our coverage tests against one interpreter, rather than every possible interpreter, ignore these branches for the sake of our branch coverage metrics. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
For some reason, Coverage thinks there's an uncovered branch when we use `next(genexpr, None)`. I think this is a Coverage bug, and I've reported it upstream, but in the meantime we can simplify some of our test functions to avoid using that construct. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
Our test for relocatable self-contained bundles has a hack in it to get the test passing when used with musl libc, which won't be exercised on most platforms. Skip test coverage for that hack. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
Cheat: improve our branch coverage by removing some branches. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
We can't assume that an interpreter is stripped if the string "stripped" appears in the output of `file`, because (depending on the version) `file` can report that a particular executable is "not stripped". Assume that it has symbols if either "stripped" doesn't appear in the output, or if "not stripped" does. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
We run some procedural code to decide which interpreters are available on the system to be tested. Since which branches are taken depends on the machine's configuration, we should exclude this test configuration code from our coverage reports. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
Some of our tests have two branches, one that's used when the interpreter is stripped and one that's used when the interpreter has symbols. Since we want the coverage tests to pass regardless of whether the interpreter being used has symbols or not, skip branch coverage for these conditionals. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
Previously we were guarding against the possibility that some lines in `/proc/$pid/maps` wouldn't match our regex, but that conditional isn't exercised, and shouldn't be possible unless our regex is incorrect. If it is incorrect, it would be better to fail the test and find out about the bad regex than to continue running the test despite the failure. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
For some reason (most likely a bug), Coverage.py reports false positive uncovered branches on lines that use generator expressions. Sidestep this by switching some tests to use list comprehensions instead of generator expressions, since Coverage handles list comprehensions correctly. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
This test had two branches, one for interpreters using 3.10 or higher, and one for interpreters using 3.9 or lower. Split it into two tests, to make the different expectations clearer and more self-documenting. Also, exclude the one testing older interpreters from coverage reports, since we expect those to only be generated using new interpreters. Signed-off-by: Matt Wozniski <mwozniski@bloomberg.net>
godlygeek
force-pushed
the
cover-tests-better
branch
from
c89e680
to
e7372b6
Compare
sarahmonod
approved these changes
May 3, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.