benchmarks: disparity dimensions mean divergence, not existence - #468
Merged
Conversation
Every one of the 19 canonical rows was flagged as having a tracked disparity, which made the flag meaningless. GaussianNB iris was the clearest casualty: fitted priors and classes exactly equal, every state diff at f32-ulp scale (theta frobenius 5e-8), score within 2.8e-8 of sklearn against a declared tolerance of 1e-3, and it still carried numerical, model-state and runtime dimensions. Two causes, both the existence-vs-divergence disease this repo has now hit three times (covered_rows > 0 in #429, the win/loss prose in #436): - "numerical" fired on any nonzero float difference. It now uses the disparity-gate precedent from #401: noise below max(1e-6, 1% of the row's declared tolerance) is not a finding. 18 rows flagged -> 3, and the three survivors are the genuine seed and tie-break gaps on the digits tree rows plus LogisticRegression. - "model-state" tested the diagnostics dict for truthiness, so the moment #402 took coverage to 19/19, every row became "disparate" by virtue of being measured. It now requires actual divergence: any first_divergent_index >= 0 (exact by construction), or a relative diff above 1e-5 whose paired absolute diff also clears 1e-7, so a near-zero denominator cannot promote pure noise. 19 rows -> 13. "runtime" is unchanged: a 37x speedup genuinely is a runtime difference, and tracking favourable differences is the report's stated purpose. Because runtime differs on every row, counts gain rows_with_substantive_disparity (dimensions beyond runtime, currently 14/19), and the benchmarks page card now shows that number with an honest label instead of the always-19/19 tracked count. The old count field is kept for compatibility; publish validation and the disparity.yml assertions were re-run against a regenerated report and pass. GaussianNB iris now reads ["runtime"] alone, which is the truth: the fitted models match to the last float digit and only the speed differs. Generated artifacts are not committed; freeze-results refreshes disparity_report.json on main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompted by the GaussianNB · iris row wearing three disparity flags while being the best-matched row in the suite: priors and classes exactly equal, every state diff at f32-ulp scale (theta frobenius 5e-8), score within 2.8e-8 against a tolerance of 1e-3.
The disease, third occurrence
All 19 rows were flagged
has_tracked_disparity, so the flag had zero discriminating power. Same existence-vs-divergence confusion ascovered_rows > 0(#429) and the win/loss prose drift (#436):max(1e-6, 1% of declared tolerance), the fix: make the disparity gate one-sided and hardware-aware (#351, #373) #401 precedent. 18 → 3, and the survivors are the genuine gaps (digits DecisionTree/RandomForest tie-break rows, LogisticRegression digits).if state:— a truthiness test on the diagnostics dict, so 19/19 the moment feat: learned-state diagnostics for every canonical benchmark row (3/19 -> 19/19) #402 completed coverage. Being measured is not being disparate. Now requires divergence: any_first_divergent_index >= 0, or relative diff > 1e-5 with paired absolute diff > 1e-7 (the pairing stops a near-zero denominator promoting noise). 19 → 13.Counts and page
Runtime differs on every row, so
rows_with_tracked_disparitystays 19/19 forever. Addedrows_with_substantive_disparity(non-runtime dimensions, currently 14/19); the benchmarks page card now shows that with an honest label. Old field kept for compatibility.Verification
["runtime"]alone.disparity.yml's assertions,publish_headline_v2.validate_disparity,model_state_coverage --require-complete, andcheck_disparity_regressionagainst the previous report all pass.node --check.freeze-resultsrefreshes the report on main.