Skip to content

Expand unit test coverage to 94%, fixing four bugs found along the way - #8

Merged
asjacobs92 merged 3 commits into
masterfrom
test/expand-unit-tests
Aug 24, 2026
Merged

Expand unit test coverage to 94%, fixing four bugs found along the way#8
asjacobs92 merged 3 commits into
masterfrom
test/expand-unit-tests

Conversation

@asjacobs92

@asjacobs92 asjacobs92 commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Coverage was 72% with four modules at 0%. This takes it to 94% — 222 tests, ~70s — and gives every module some.

Module Before After
utils/persist.py 0% 92%
utils/rootpath.py 0% 98%
utils/log.py 0% 100%
utils/const.py 0% 100%
report/plot.py 56% 96%
utils/plot.py 84% 95%
report/trust.py 78% 88%
utils/dataset.py 67% 98%
main.py 90% 97%
Total 72% 94%

Bugs the new tests found

Writing tests for the untouched paths turned up four real defects — two of them upgrade fallout that survived 1.2.0 because nothing exercised the code.

np.uint8(-1) raises OverflowError on NumPy 2. It wrapped silently to 255 on NumPy 1, so cic_ids_2017_label_converter() blew up on any unrecognised label instead of returning its sentinel. 255 is now explicit, preserving the old result.

DataFrame.groupby(axis=1) was removed in pandas 2.0, so plot_distribution(aggregate=True) raised TypeError. Rewritten to group bit columns by prefix explicitly, preserving column order so the bits keep their significance. I verified the arithmetic separately rather than just asserting it runs — flags_0/flags_1 fold to [1, 2, 3, 0].

class_names defaults to [] at seven indexing sites guarded only by is not None — which passes for an empty list. Calling any of those entry points without class_names raised IndexError, as did a class_names shorter than the tree's class count. Added a _class_label() helper doing the bounds check trust.py already did.

labels[i] was indexed unguarded in plot_stacked_bars, while every other site in that file bounds-checks.

Behaviour documented, not changed

Three surprises I left alone but pinned with tests, since changing them is a judgement call:

  • rootpath.detect() returns None at the first empty directory it meets — an empty subdirectory aborts the search before reaching the marker above it.
  • plot_distribution(aggregate=True) requires feature_names; it converts X with .values, so DataFrame column names never reach the prefix regex.
  • skip_retrain=True reuses the blackbox as-is and therefore requires an already-fitted model.

Flakiness

Trustee.fit() samples via np.random.choice and calls train_test_split without a random_state — both draw on NumPy's global RNG. My first fidelity assertion passed, then failed on a later run. An autouse fixture now seeds per test; I ran the full suite three times over to confirm it's stable.

Worth flagging separately: the library itself is nondeterministic by default. Trustee(...).fit() gives different explanations across runs unless the caller seeds NumPy globally. That may be intended for the stability analysis, but there's no random_state parameter to pin it. Happy to add one if you want it.

Also

  • Every plotting test asserts no matplotlib figure is left open, guarding the leak fixed in 381eb17.
  • CI gains --cov-fail-under=90 so coverage can't quietly regress.
  • One narrow warning allowlist: joblib 1.5.3 (already latest) sets ndarray.shape directly, which NumPy 2.5 deprecated. It fires on every load_model() and there's no version to upgrade to. Scoped to the exact message so DeprecationWarning stays an error otherwise — worth watching, since it becomes a hard break when NumPy removes it.

asjacobs92 and others added 3 commits August 24, 2026 18:16
Coverage was 72% with four modules at 0%. This takes it to 94% (222 tests,
~70s) and covers every module that had none.

New suites:

* test_persist.py  -- save/load round trip, the zip path and its cleanup,
  and the not-a-model / missing-path cases.
* test_rootpath.py -- marker detection from files and nested directories,
  custom patterns, and the empty-directory quirk (see below).
* test_log.py      -- message formatting, levels, filtering and handlers.
* test_const.py    -- structural validation of the bundled dataset metadata
  (read() unpacks each field as a 4-tuple, so a malformed entry would only
  surface as an opaque error at read time) plus the label converter.
* test_utils_plot.py / test_report_plot.py -- every plotting entry point,
  each asserting no matplotlib figure is left open. That guards the leak
  fixed in 381eb17 and caught two of the bugs below.

Bugs found by the new tests:

* trustee/utils/const.py: np.uint8(-1) raises OverflowError on NumPy 2 --
  NumPy 1 wrapped it to 255. cic_ids_2017_label_converter() therefore blew up
  on any unrecognised label instead of returning its sentinel. 255 is now the
  explicit sentinel, preserving the NumPy 1 result.
* trustee/report/plot.py: DataFrame.groupby(axis=1) was removed in pandas 2.0,
  so plot_distribution(aggregate=True) raised TypeError. Rewritten to group
  the bit columns by prefix explicitly, preserving column order so the bits
  keep their significance. This one survived the 1.2.0 upgrade because
  nothing exercised the path.
* trustee/report/plot.py: class_names defaults to [] at seven indexing sites
  guarded only by `is not None`, which passes for an empty list. Calling any
  of those entry points without class_names raised IndexError, as did a
  class_names shorter than the tree's class count. Added a _class_label()
  helper doing the bounds check trust.py already did.
* trustee/utils/plot.py: labels[i] was indexed unguarded in plot_stacked_bars
  while every other site in the file bounds-checks.

Behaviour documented rather than changed:

* rootpath.detect() returns None on the first empty directory it meets, so an
  empty subdirectory aborts the search before reaching the marker above it.
* plot_distribution(aggregate=True) needs feature_names -- it converts X with
  .values, so DataFrame column names never reach the prefix regex.
* skip_retrain=True reuses the blackbox as-is and so requires a fitted model.

Determinism: Trustee.fit() samples via np.random.choice and calls
train_test_split without a random_state, both drawing on NumPy's global RNG,
which made any assertion on fidelity flaky. An autouse fixture now seeds it
per test; the suite was run three times over to confirm.

CI gains --cov-fail-under=90 so coverage cannot quietly regress.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
os.path.expanduser resolves ~ from USERPROFILE on Windows (ntpath checks it
first and never consults HOME), so setting only HOME left ~ pointing at the
real profile directory, which has no root marker. Set both rather than
skipping the test on one platform.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds a `coverage-comment` job that posts a coverage summary to the PR once
every test leg has passed.

* `needs: test` gates it on the whole matrix, so a comment only ever appears
  for a run whose tests actually passed.
* The report is rendered from coverage.json produced by the ubuntu/3.12 leg,
  which alone uploads the artifact -- otherwise six matrix legs race to write
  the same artifact name.
* The comment is located by a hidden marker and updated in place, so pushing
  to a PR revises one comment instead of stacking a new one per run. The
  lookup matches on the marker *and* on github-actions[bot] as the author, so
  a comment that merely quotes the marker is never edited.
* Skipped for forks, whose GITHUB_TOKEN is read-only and cannot comment.

.github/scripts/coverage_summary.py renders the Markdown: overall percentage
against the floor, a bar, and a table of partially covered files sorted
worst-first with their uncovered line ranges condensed. Fully covered files
collapse into a <details> line rather than padding the table.

Posting uses the gh CLI already on the runner rather than another third-party
action, and the jq lookup was checked against the live API and against
fixtures covering the match, no-match and impostor-author cases.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

Coverage report

93.51% meets the 90% floor.

███████████████████░ 1,483 / 1,586 statements · 103 uncovered

File Coverage Missed Uncovered lines
trustee/report/trust.py 88% 71 740, 745, 756, 770–771, 780, 830, +42 more
trustee/utils/persist.py 92% 2 27–28
trustee/utils/plot.py 95% 10 38, 129, 178, 268, 274, 353, +3 more
trustee/report/plot.py 96% 11 232, 522, 570–571, 608–612, 618–619
trustee/main.py 97% 5 217, 269, 509, 536, 553
trustee/utils/rootpath.py 98% 1 104
trustee/utils/dataset.py 98% 2 86, 116
trustee/utils/tree.py 99% 1 28
5 file(s) at 100%

trustee/__init__.py, trustee/_version.py, trustee/enums/feature_type.py, trustee/utils/const.py, trustee/utils/log.py

Measured on Python 3.12, ubuntu-latest.

@asjacobs92
asjacobs92 merged commit 7357580 into master Aug 24, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant