fix(eval): stop zero-variance Cohen's d from reporting no effect - #39
Merged
aurascoper merged 1 commit intoJul 27, 2026
Merged
Conversation
cohens_d returned 0.0 whenever pooled standard deviation was zero, regardless
of whether the means were identical or completely separated. Two perfectly
separated constant groups therefore reported "no effect" — the least correct
available answer, and one that ranks alongside genuine null results.
Zero pooled variance now distinguishes the two cases:
equal constant groups 0.0 a true zero
separated constant groups NaN undefined, not absent
NaN rather than infinity: d is undefined without within-group variation, NaN
does not pretend the effect is absent, it matches the function's existing
under-sized-sample behaviour, and it forces callers to treat the estimate as
degenerate instead of ranking it as an ordinary finite effect.
Finishes a consolidation the module already claimed. eval_stats.py's docstring
says it consolidates cohens_d "previously duplicated in embedding_analyze.py
and statistical_analysis.py", but the duplicates were never removed — and
analyze_results.py had a fourth copy. All three now import the shared
implementation, following the established `sys.path.insert` + `from eval_stats
import ...` idiom already used for bootstrap_ci. Fixing one copy would
otherwise have left three wrong ones.
test_cohens_d_large_effect conflated "large" with "zero-variance degenerate";
it is replaced by three tests that separate identical-constant, separated-
constant, and large-non-degenerate cases.
Removes the first of three deselections in the python-contracts job (issue #38).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of three PRs resolving #38. Removes exactly one CI deselection.
The defect
cohens_dreturned0.0whenever pooled SD was zero, regardless of whether the means were identical or completely separated:Zero pooled variance now distinguishes the two cases:
0.00.0(a true zero)0.0NaN(undefined)NaNNaN(unchanged)NaNover infinity, per the issue: d is undefined without within-group variation,NaNdoesn't pretend the effect is absent, it matches the function's existing under-sized-sample behaviour, and it forces callers to treat the estimate as degenerate rather than ranking it as an ordinary finite effect.Consolidation, because there were four copies
eval_stats.py's docstring already claims it consolidatescohens_d"previously duplicated in embedding_analyze.py and statistical_analysis.py" — but the duplicates were never removed, andanalyze_results.pyheld a fourth. All four were byte-identical in logic, so fixing one would have left three wrong ones.analyze_results.py,embedding_analyze.py, andstatistical_analysis.pynow import the shared implementation, following thesys.path.insert(0, str(Path(__file__).parent))idiom already used in this directory forbootstrap_ci. Verified at runtime that all three resolve to the same function object.Scope note:
mann_whitney_uis also duplicated across these scripts. Left alone deliberately — that is 38-B's subject and does not belong in this diff.Tests
test_cohens_d_large_effectconflated "large" with "zero-variance degenerate". Replaced by three tests that separate the cases:Expected CI counts
The issue projected
84 passed, 2 deselected. Actual will be 88 collected / 2 deselected / 86 passed: this PR replaces one test with three, so the collection grows by two. Nothing was skipped — the progression is 83 → 86 passed because the suite gained tests alongside the fix.Local:
test_eval_stats9 passed, 1 failed (the still-deselected Mann-Whitney case, 38-B's subject). py_compile, YAML parse, andgit diff --checkall clean.