Skip to content

fix(coverage): --coverage-diff passes a new file that no test executed #1054

Description

@Chemaclass

Problem

--coverage-diff reports 100% for a brand new file that no test executed, and --coverage-min passes. That is the exact case the gate exists to catch.

report_diff (src/coverage/diff.sh:102) iterates get_tracked_files, which only holds files that executed at least once. A changed file that never ran is skipped, the changed-line total stays 0, and diff_percentage returns 100 for an empty set (src/coverage/diff.sh:37-41, intentionally, so a docs-only commit does not fail).

Reproduced on Bash 3.2 arm64, macOS. Added an untracked file with three executable lines and no test:

printf '#!/usr/bin/env bash\n\nfunction never_called_probe() {\n  echo "one"\n  echo "two"\n  echo "three"\n}\n' > src/zz_probe_tmp.sh

./bashunit --coverage --coverage-paths src --coverage-diff HEAD --coverage-min 90 \
  --coverage-report /dev/null tests/unit/assert/basic_test.sh

Output:

Diff Coverage (vs HEAD)
---------------
No changed executable lines.
---------------
Total: 0/0 (100%)

Exit code 0. A PR adding a fully untested file passes a 90% diff gate.

The empty-set-is-100% rule is correct on its own. The defect is the input set: "nothing changed" and "the changed file never ran" are being treated as the same thing.

Proposal

Compute diff coverage over the changed files, not over the executed files.

  • Ask git for the changed files against the base ref, filter them through the coverage paths and excludes, and iterate that set unioned with the tracked set.
  • A changed file with no hit data yields 0/N, not a skip.
  • Keep the empty-set-is-100% rule for the case it was written for: no changed executable lines anywhere.
  • bashunit::helper::git_changed_lines (src/helper/git.sh:101) already handles an untracked file by treating every line as new, so the per-file half works once the file reaches the loop.

Seeding tracked files from the coverage paths would fix this as a side effect, but the diff report should not depend on that: it needs the changed set regardless, and a changed file outside the seeded paths must still be excluded.

Where to change

  • src/coverage/diff.sh:91-137 report_diff
  • src/coverage/stats.sh:164-186 check_threshold, which reads _BASHUNIT_COVERAGE_DIFF_PCT_OUT
  • src/helper/git.sh:137 git_filter_changed is the existing helper for the changed-file list

Acceptance criteria

  • A new, never-executed file with changed executable lines appears in the diff report as 0/N (0%)
  • --coverage-diff <ref> --coverage-min 90 exits non-zero for that file
  • An untracked (new, uncommitted) file is covered by the same behaviour, matching git_changed_lines
  • A modified file whose changed lines all ran still reports 100%
  • A run with no changed executable lines anywhere still reports 100% and passes the gate (feat(coverage): diff coverage — restrict the report to lines changed against a base ref #1032 behaviour preserved)
  • A changed file outside BASHUNIT_COVERAGE_PATHS, or matched by BASHUNIT_COVERAGE_EXCLUDE, is not counted
  • A changed file that was deleted does not break the report
  • A renamed file resolves to its new path, matching --changed (feat(cli): --changed to run only the tests touched since a git ref #1010)
  • Works under --parallel
  • Both engines produce the same numbers (trap and xtrace)
  • LCOV and HTML stay whole-file, as documented in src/coverage/diff.sh:10-12

Repo checklist (agent)

  • TDD: RED then GREEN then REFACTOR. The RED test is the reproduction above, in a fixture git repo under a temp dir, never against this repo's own history.
  • Bash 3.0+ only: no printf -v, no += append, no declare -A, no [[ ]], no ${var,,}, no &>>, no ${arr[-1]}. Expanding a possibly-empty array under set -u needs ${arr[@]+"${arr[@]}"}.
  • Fixtures under tests/acceptance/fixtures/ must not end in *test.sh.
  • Gates: make sa, make lint, ./bashunit tests/, ./bashunit --parallel tests/. Never run shfmt -w.
  • CHANGELOG.md: one line under ## Unreleased (Fixed).
  • One issue = one PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions