Skip to content

fix: correct patch coverage calculation and baseline artifact lookup#56

Merged
MathurAditya724 merged 2 commits intomainfrom
fix/failed-jobs-can-also-be-used
Mar 24, 2026
Merged

fix: correct patch coverage calculation and baseline artifact lookup#56
MathurAditya724 merged 2 commits intomainfrom
fix/failed-jobs-can-also-be-used

Conversation

@MathurAditya724
Copy link
Copy Markdown
Member

Summary

Fixes incorrect patch coverage reporting (100% instead of actual value) caused by two independent bugs, plus improves default behavior for PR comments.

Context: Observed in pydantic/monty#288 where the codecov-action reported Patch coverage is 100.00% while the official Codecov bot correctly reported 61.01%.

Root Causes & Fixes

1. Path mismatch in patch coverage calculation (primary fix)

Coverage tools like cargo llvm-cov emit absolute paths (e.g., /home/runner/work/monty/monty/crates/monty/src/args.rs) while PR diffs use repo-relative paths (crates/monty/src/args.rs). The patch analyzer only did exact Map.get() lookups, so no files ever matched, resulting in 0/0 = 100%.

Fix: Added a 3-tier path resolution strategy in PatchAnalyzer:

  1. Exact match (existing)
  2. Normalized equality (new — handles ./src/file.ts vs src/file.ts)
  3. Suffix-based matching (new — handles absolute vs relative paths)

Results are cached per diff path for performance. Also replaced O(n) Array.find() line lookups with O(1) Map.get().

2. Baseline artifact lookup too restrictive

The artifact download filtered workflow runs by status: "success", but in repos like pydantic/monty, the overall workflow conclusion is "failure" due to unrelated jobs failing — even when the job that uploads the coverage artifact succeeded.

Fix: Changed to status: "completed" with post-filter on conclusion to accept "success" and "failure" (excluding "cancelled", "timed_out", etc. which may have incomplete artifacts).

3. Default filesMode changed from "all" to "changed"

PR comments were showing all 89 project files with uncovered lines instead of just the 4 changed files. Now defaults to showing only changed files unless explicitly overridden.

4. Clear warning on path mismatch

When no diff files match coverage data (0/0 case), a core.warning() is now emitted with sample paths from both sides to help users diagnose the issue.

5. Bidirectional suffix matching in report-formatter

The file list filter in the report formatter now checks suffix matching in both directions (coverage→diff and diff→coverage) for consistency with the patch analyzer.

Files Changed

File Change
src/analyzers/patch-analyzer.ts Suffix path matching, O(1) line lookups, unmatched file tracking, warning on 0/0
src/utils/artifact-manager.ts status: "completed" + conclusion filter
src/config/config-loader.ts Default filesMode"changed"
src/formatters/report-formatter.ts Default filesMode"changed", bidirectional suffix match
src/index.ts Updated effectiveFilesMode logic
src/__tests__/patch-analyzer.test.ts Tests for suffix matching, unmatched files
src/__tests__/config-loader.test.ts Updated default expectations
src/__tests__/report-formatter.test.ts Tests for reverse suffix match, default mode

Testing

  • All 177 tests pass
  • New test cases cover:
    • Absolute coverage paths matched against relative diff paths
    • Unmatched file tracking when no coverage paths match
    • Reverse suffix matching (absolute diff paths, relative coverage paths)
    • Default filesMode behavior

Comment thread src/index.ts Outdated
- Add suffix-based path matching in PatchAnalyzer to handle absolute vs
  relative path mismatches (e.g. cargo llvm-cov emits absolute paths while
  PR diffs use repo-relative paths)
- Change artifact lookup from status:'success' to status:'completed' with
  conclusion filtering (success/failure), so base coverage artifacts from
  workflow runs that failed due to unrelated jobs are still found
- Default filesMode to 'changed' so PR comments only show files touched
  in the PR rather than all project files with uncovered lines
- Add bidirectional suffix matching in report-formatter file filter for
  consistency with the patch analyzer
- Add clear warning when no diff files match coverage data (0/0 = 100%)
- Replace O(n) line lookups with Map-based O(1) lookups in patch analyzer
- Add normalized equality check between exact match and suffix match

Fixes: getsentry/codecov-action reporting 100% patch coverage when
coverage tools emit absolute file paths (observed in pydantic/monty#288)
@MathurAditya724 MathurAditya724 force-pushed the fix/failed-jobs-can-also-be-used branch from c29fa12 to d042d2e Compare March 24, 2026 18:15
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 24, 2026

Codecov Results 📊

179 passed | Total: 179 | Pass Rate: 100% | Execution Time: 239ms

📊 Comparison with Base Branch

Metric Change
Total Tests 📈 +5
Passed Tests 📈 +5
Failed Tests
Skipped Tests

All tests are passing successfully.

❌ Patch coverage is 65.00%. Project has 757 uncovered lines.
✅ Project coverage is 55.94%. Comparing base (base) to head (head).

Files with missing lines (5)
File Patch % Lines
index.ts 0.00% ⚠️ 310 Missing
artifact-manager.ts 0.00% ⚠️ 155 Missing
report-formatter.ts 44.27% ⚠️ 143 Missing and 10 partials
config-loader.ts 94.23% ⚠️ 2 Missing and 5 partials
patch-analyzer.ts 97.14% ⚠️ 2 Missing and 2 partials
Coverage diff
@@            Coverage Diff             @@
##          main       #PR       +/-##
==========================================
+ Coverage    55.46%    55.94%    +0.48%
==========================================
  Files           24        24         —
  Lines         1679      1713       +34
  Branches      1208      1232       +24
==========================================
+ Hits           929       956       +27
- Misses         750       757        +7
- Partials        95        97        +2

Generated by Codecov Action

…ranch

On feature branch pushes (non-PR), patchCoverage is null so changedFiles
is empty, causing the file table to be hidden entirely. The previous
check only fell back to 'all' when currentBranch === baseBranch, but
feature branches also lack PR context and need the same fallback.
@MathurAditya724 MathurAditya724 merged commit 861b96c into main Mar 24, 2026
9 checks passed
@MathurAditya724 MathurAditya724 deleted the fix/failed-jobs-can-also-be-used branch March 24, 2026 18:25
Comment on lines +75 to +76
normalizedCoverage.endsWith(`/${normalizedDiff}`) ||
normalizedDiff.endsWith(`/${normalizedCoverage}`)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The endsWith() check for matching coverage files can cause false positives by incorrectly matching files that share a common path suffix in different directories.
Severity: MEDIUM

Suggested Fix

The matching logic should be made more robust. Instead of a simple endsWith() check, consider a more precise matching algorithm that verifies the full path segments. For example, ensure that the character preceding the matched suffix is a path separator ('/') or that the paths are identical.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: src/analyzers/patch-analyzer.ts#L75-L76

Potential issue: The patch coverage calculation in `findCoverageFile()` uses a simple
`endsWith()` check to match changed files with files in the coverage report. This can
lead to incorrect matches in monorepos where different files share a common path suffix.
For example, a changed file `lib/file.ts` could be incorrectly associated with coverage
data from `src/lib/file.ts` because `normalizedCoverage.endsWith('/' + normalizedDiff)`
evaluates to true. This results in inaccurate patch coverage calculations by attributing
coverage to the wrong file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant