Three pre-existing Tests/eval failures deselected by the new Python CI job

[aurascoper]

Three tests in Tests/eval fail on docs/eeg-methods-scope. They are
pre-existing, reproduce identically on NumPy 1.26.4 and 2.4.6, and surfaced only
because #37 puts these suites under CI for the first time. #37 deselects them by
name so the other 59 tests can gate; this issue tracks resolving them.

My reading is that two are wrong tests and one is a real defect, but all
three are statistical judgement calls, so none were changed unilaterally.

1. test_cohens_d_large_effect — likely a real defect

cohens_d([1, 1, 1, 1], [5, 5, 5, 5])  # returns 0.0, test asserts > 2.0

Evaluation/scripts/analyze_results.py:48:

if pooled_std == 0:
    return 0.0

Both groups are constant, so pooled SD is 0 and Cohen's d is undefined
(unbounded). Returning 0.0 reports "no effect" for two perfectly separated
groups — the most misleading available sentinel. The function already returns
nan for the other degenerate case (len < 2), so nan would be consistent,
and inf would be defensible.

Duplicated in Evaluation/scripts/embedding_analyze.py:57, so any fix should
cover both.

The test input is degenerate, so the test could reasonably be rewritten too —
but a zero-variance guard that silently claims no effect is worth fixing on its
own merits, since real evaluation data can produce constant groups.

2. test_mann_whitney_u_disjoint — the test asks for the impossible

mann_whitney_u([1, 2, 3], [10, 11, 12])  # p = 0.1, test asserts < 0.05

With n₁ = n₂ = 3 the smallest attainable two-sided p is 2/20 = 0.1, which is
what the implementation returns for completely disjoint groups. No correct
implementation can pass this assertion at that sample size. The fix is in the
test: raise n to at least 4 per group, or assert p <= 0.1.

3. test_cka_independent — the expectation ignores estimator bias

X = randn(50, 128); Y = randn(50, 256)
cka(X, Y)  # 0.781, test asserts < 0.3

Evaluation/scripts/embedding_space_analysis.py:37 implements standard
biased linear CKA (Kornblith et al. 2019) and centers correctly. The biased
estimator is known to be inflated when dimensionality exceeds sample count, and
here d ≫ n (128 and 256 versus 50), so ~0.78 on independent data is expected
behaviour rather than a bug.

Two ways out, and they answer different questions:

• keep the biased estimator and fix the test — raise n well above d, or relax
  the threshold to reflect the bias at these shapes;
• switch to unbiased CKA, which is the better default when the tool is used to
  compare representations of differing width, as the test itself does.

The second changes reported numbers in any existing analysis, so it is a
research decision, not a cleanup.

Reproduce

pip install pytest numpy pandas scipy
pytest Tests/eval/test_eval_stats.py Tests/eval/test_embedding_space.py -v

Closing condition

All three pass, and the --deselect lines are removed from the
python-contracts job in .github/workflows/ci.yml.

[aurascoper]

Resolution opened as three stacked PRs, one deselection each:

PR	subject	disposition
#39	zero-variance Cohen's d	implementation defect — fixed
#40	Mann–Whitney at n=3	invalid test expectation — test corrected
#41	biased CKA null at d ≫ n	invalid test expectation — both regimes pinned

Merge order is #39 → #40 → #41; each is based on the previous and will need retargeting to docs/eeg-methods-scope as its parent lands.

Two notes on the diagnosis in this issue, both confirmed against the code:

Cohen's d was duplicated four times, not once. eval_stats.py's docstring already claims it consolidates cohens_d "previously duplicated in embedding_analyze.py and statistical_analysis.py" — but the duplicates were never removed, and analyze_results.py held a fourth. All four were logically identical, so #39 finishes the consolidation rather than patching one copy and leaving three wrong.

The projected counts shift upward. The issue projected 84 → 85 → 86. Actual will be higher because two of these PRs add tests while fixing: #39 replaces one test with three, #41 replaces one with two. Expect roughly 86 → 87 → 89, with zero deselections at the end. The gate strengthens by more than the deselection count suggests.

Closing condition remains as stated: CI green with no --deselect flags in the python-contracts job. #41 removes the last one.

[aurascoper]

Resolved. All three PRs merged into docs/eeg-methods-scope:

#39  b8894ee8  zero-variance Cohen's d       86 passed, 2 deselected
#40  c064bcb8  attainable Mann-Whitney       87 passed, 1 deselected
#41  41f559a8  biased CKA null regimes       89 passed, 0 deselected

Closing condition met and verified against the merged tip (41f559a) rather than any single PR's pre-merge run:

grep -c deselect .github/workflows/ci.yml          0
pytest Tests/eval --collect-only                   89 tests collected
pytest Tests/eval/test_eval_stats.py \
       Tests/eval/test_embedding_space.py          20 passed

The python-contracts job now runs the eval directory with no --deselect flags. The only remaining exclusion is test_joint_embedding.py, which imports mlx.core and cannot run on a Linux runner — unrelated to this issue and documented in the workflow.

The gate went from 86 tests with 3 excluded to 89 with none, so it is stronger by more than the three restored cases: #39 and #41 each added tests while fixing.

Dispositions held as diagnosed — one implementation defect, two invalid test expectations:

Cohen's d returned 0.0 for two perfectly separated zero-variance groups. Now 0.0 only for equal constant groups and NaN when separated, matching its existing under-sized-sample behaviour. One correction to this issue's text: the duplication was four copies, not three — eval_stats.py itself was the fourth, and its docstring already claimed a consolidation that had never been completed. All three other scripts now import the shared implementation.

Mann-Whitney asserted p < 0.05 at n=3 per group, where the exact two-sided minimum is 2/20 = 0.1. Implementation was correct; the test was unsatisfiable. Four per group gives 0.0286.

CKA asserted < 0.3 for independent data at n=50, d=128/256, where the biased estimator scores ~0.785. The estimator is unchanged deliberately — switching to unbiased CKA would alter prior representation-comparison results and needs a registered migration. Both regimes are now pinned (0.042 at n≫d, 0.785 at d≫n) and cka() documents the inflation.