Skip to content

fix(ci): resolve test count inconsistencies in reporting scripts#519

Merged
hhimanshu merged 2 commits into
mainfrom
fix/test-count-reporting
Apr 25, 2026
Merged

fix(ci): resolve test count inconsistencies in reporting scripts#519
hhimanshu merged 2 commits into
mainfrom
fix/test-count-reporting

Conversation

@hhimanshu
Copy link
Copy Markdown
Contributor

Summary

  • Removes the hardcoded 65 properties × 500 cases = 32,500 line from post-conformance-comment.sh — there are now 132 property test functions (×500 = 66,000 cases), so the old number was wrong. The breakdown already appears correctly in the coverage comment which reads junit.xml dynamically.
  • Adds nextest_total to post-coverage-comment.sh and generate-pages-html.py by summing testsuite/@tests from junit.xml at runtime.
  • Adds an "About the totals" note to both the PR coverage comment footer and the GitHub Pages explainer, clarifying that the ~76,988 grand total counts formula evaluations while the GitHub Checks badge reports Rust #[test] functions — two intentionally different metrics.

Closes #518

Before / After

Before: conformance PR comment said "32,500 property inputs"; coverage comment said "66,000 (132×500)". Pages had no explanation for why Checks showed 2,879.

After: conformance comment no longer mentions property tests (the coverage comment covers it); coverage comment footer and Pages explainer both state the nextest count and explain the difference.

Test plan

  • bash -n passes on both shell scripts
  • python3 -c "import ast; ast.parse(...)" passes on generate-pages-html.py
  • CI green on this PR

🤖 Generated with Claude Code

- Remove hardcoded "65 properties × 500 cases = 32,500" from
  post-conformance-comment.sh; the property breakdown is already shown
  correctly in the coverage comment (which reads junit.xml dynamically)
- Add nextest_total to post-coverage-comment.sh and generate-pages-html.py
  by summing testsuite/@tests from junit.xml
- Add an "About the totals" note to both the PR coverage comment and the
  GitHub Pages explainer clarifying that the ~N grand total counts formula
  evaluations while GitHub Checks reports Rust #[test] functions — two
  intentionally different metrics

Closes #518

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hhimanshu hhimanshu self-assigned this Apr 25, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 25, 2026

Test Coverage by Category

Category Unit Tests Google Sheets Conformance Property Cases Total
Array 42 301/301 ✓ 1,000 (2×500) 1,343
Database 35 175/175 ✓ 3,500 (7×500) 3,710
Date 348 295/295 ✓ 2,500 (5×500) 3,143
Engineering 245 841/841 ✓ 5,500 (11×500) 6,586
Filter 11 20/20 ✓ 4,500 (9×500) 4,531
Financial 149 182/182 ✓ 2,000 (4×500) 2,331
Info 0 390/390 ✓ 4,500 (9×500) 4,890
Logical 114 250/250 ✓ 3,500 (7×500) 3,864
Lookup 69 361/361 ✓ 1,000 (2×500) 1,430
Math 528 1,923/1,923 ✓ 8,000 (16×500) 10,451
Operator 75 263/263 ✓ 7,500 (15×500) 7,838
Parser 80 94/94 ✓ 4,000 (8×500) 4,174
Statistical 480 2,557/2,557 ✓ 5,000 (10×500) 8,037
Text 305 658/660 ⚠ 4,000 (8×500) 4,965
Volatile 0 3,500 (7×500) 3,500
Web 29 56/56 ✓ 6,000 (12×500) 6,085
Total 2,620 8,366/8,368 66,000 (132×500) ~76,988

Oracle: Google Sheets · ✓ = 100% passing · ⚠ = known deviation · The ~76,988 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 2,879 Rust test functions.

…ology

Test count clarity:
- post-coverage-comment.sh and generate-pages-html.py now compute the
  nextest breakdown dynamically: unit + property functions + conformance/
  integration, so readers can see exactly how 2,620 + 159 + 100 = 2,879
- Drop "Oracle: Google Sheets" footer prefix from all reporting surfaces;
  replace with "Google Sheets conformance" or plain "Google Sheets"

Remove "oracle" from the codebase:
- Replace // oracle: inline test annotations with // gs: in src/
- Replace "Oracle helpers" comment in conformance_reporter.rs
- Replace "Oracle artifact" comment in conformance.rs
- Replace "oracle fixtures" in property_conformance.rs
- Update CLAUDE.md section 8 heading and body
- Rename xtask/src/oracle_sheets.rs → sheets_eval.rs, SheetsOracle →
  SheetsEvaluator, OracleEvaluate command → GsEvaluate, GAS_ORACLE_URL →
  GAS_URL, run_oracle_evaluate → run_gs_evaluate

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 25, 2026

Test Coverage by Category

Category Unit Tests Google Sheets Conformance Property Cases Total
Array 42 301/301 ✓ 1,000 (2×500) 1,343
Database 35 175/175 ✓ 3,500 (7×500) 3,710
Date 348 295/295 ✓ 2,500 (5×500) 3,143
Engineering 245 841/841 ✓ 5,500 (11×500) 6,586
Filter 11 20/20 ✓ 4,500 (9×500) 4,531
Financial 149 182/182 ✓ 2,000 (4×500) 2,331
Info 0 390/390 ✓ 4,500 (9×500) 4,890
Logical 114 250/250 ✓ 3,500 (7×500) 3,864
Lookup 69 361/361 ✓ 1,000 (2×500) 1,430
Math 528 1,923/1,923 ✓ 8,000 (16×500) 10,451
Operator 75 263/263 ✓ 7,500 (15×500) 7,838
Parser 80 94/94 ✓ 4,000 (8×500) 4,174
Statistical 480 2,557/2,557 ✓ 5,000 (10×500) 8,037
Text 305 658/660 ⚠ 4,000 (8×500) 4,965
Volatile 0 3,500 (7×500) 3,500
Web 29 56/56 ✓ 6,000 (12×500) 6,085
Total 2,620 8,366/8,368 66,000 (132×500) ~76,988

✓ = 100% passing · ⚠ = known deviation · The ~76,988 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 2,879 Rust test functions: 2,620 unit + 159 property functions (shown as cases above) + 100 conformance/integration.

@hhimanshu hhimanshu merged commit 837ab06 into main Apr 25, 2026
2 checks passed
@hhimanshu hhimanshu deleted the fix/test-count-reporting branch April 25, 2026 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(ci): resolve test count inconsistencies across GitHub Pages, PR comments, and Checks

1 participant