fix(ci): resolve test count inconsistencies in reporting scripts#519
Merged
Conversation
- Remove hardcoded "65 properties × 500 cases = 32,500" from post-conformance-comment.sh; the property breakdown is already shown correctly in the coverage comment (which reads junit.xml dynamically) - Add nextest_total to post-coverage-comment.sh and generate-pages-html.py by summing testsuite/@tests from junit.xml - Add an "About the totals" note to both the PR coverage comment and the GitHub Pages explainer clarifying that the ~N grand total counts formula evaluations while GitHub Checks reports Rust #[test] functions — two intentionally different metrics Closes #518 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Test Coverage by Category
Oracle: Google Sheets · ✓ = 100% passing · ⚠ = known deviation · The ~76,988 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 2,879 Rust test functions. |
…ology Test count clarity: - post-coverage-comment.sh and generate-pages-html.py now compute the nextest breakdown dynamically: unit + property functions + conformance/ integration, so readers can see exactly how 2,620 + 159 + 100 = 2,879 - Drop "Oracle: Google Sheets" footer prefix from all reporting surfaces; replace with "Google Sheets conformance" or plain "Google Sheets" Remove "oracle" from the codebase: - Replace // oracle: inline test annotations with // gs: in src/ - Replace "Oracle helpers" comment in conformance_reporter.rs - Replace "Oracle artifact" comment in conformance.rs - Replace "oracle fixtures" in property_conformance.rs - Update CLAUDE.md section 8 heading and body - Rename xtask/src/oracle_sheets.rs → sheets_eval.rs, SheetsOracle → SheetsEvaluator, OracleEvaluate command → GsEvaluate, GAS_ORACLE_URL → GAS_URL, run_oracle_evaluate → run_gs_evaluate Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Test Coverage by Category
✓ = 100% passing · ⚠ = known deviation · The ~76,988 total counts formula evaluations (each conformance row and each property case = 1). GitHub Checks reports 2,879 Rust test functions: 2,620 unit + 159 property functions (shown as cases above) + 100 conformance/integration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
65 properties × 500 cases = 32,500line frompost-conformance-comment.sh— there are now 132 property test functions (×500 = 66,000 cases), so the old number was wrong. The breakdown already appears correctly in the coverage comment which readsjunit.xmldynamically.nextest_totaltopost-coverage-comment.shandgenerate-pages-html.pyby summingtestsuite/@testsfromjunit.xmlat runtime.~76,988grand total counts formula evaluations while the GitHub Checks badge reports Rust#[test]functions — two intentionally different metrics.Closes #518
Before / After
Before: conformance PR comment said "32,500 property inputs"; coverage comment said "66,000 (132×500)". Pages had no explanation for why Checks showed 2,879.
After: conformance comment no longer mentions property tests (the coverage comment covers it); coverage comment footer and Pages explainer both state the nextest count and explain the difference.
Test plan
bash -npasses on both shell scriptspython3 -c "import ast; ast.parse(...)"passes ongenerate-pages-html.py🤖 Generated with Claude Code