Skip to content

ci: run the Python contract suites - #37

Merged
aurascoper merged 2 commits into
docs/eeg-methods-scopefrom
ci/python-contract-suites
Jul 27, 2026
Merged

ci: run the Python contract suites#37
aurascoper merged 2 commits into
docs/eeg-methods-scopefrom
ci/python-contract-suites

Conversation

@aurascoper

Copy link
Copy Markdown
Owner

Stacked on #36merge that first. test_session_consume imports eeg_spectral, which cannot import on numpy>=2 without it, so this job would be red on its own.

Prerequisite for #33 going ready, per the agreed gate github_ci_runs_tests_eval: true.

Why nothing caught the missing module

build-and-test is Swift-only. Neither NeuralComposeEEG/tests nor Tests/eval has ever run in CI, leaving 62 Python tests ungated — which is exactly how #33's test could import an uncommitted module and still show a green check.

pytest is required here, not preferred

Tests/eval mixes four harness styles. Most modules are bare def test_* functions with no TestCase and no __main__ runner:

module python -m unittest collects real tests
test_dialectic_corpus_quarantine 2 2
test_eeg_spectral_compat 6 6
test_eeg_channel_quality 0 6
test_embedding_space 0 9
test_eval_stats 0 8
test_overnight_watchdog 0 8
test_state_reconstruction 0 4
test_state_trajectory_analysis 0 5

unittest prints OK for every one of those zero-collection modules. My first draft of this job used unittest and would have shipped a green check that ran 8 of 62 tests. pytest collects all 62.

Exclusions, each with a reason

test_joint_embedding.py imports mlx.core (Apple Silicon only) — run locally.

Three cases are deselected because they already fail on this base, on both NumPy 1.26.4 and 2.4.6. They are not caused by this PR and are not silently dropped — see the tracking issue. The remaining tests in those files still run.

Result

NeuralComposeEEG contracts   33 passed   (unittest discover; grows to 46 once #35 lands)
Tests/eval                   59 passed, 3 deselected

torch installs from the CPU wheel index — the default Linux wheel is a ~800MB CUDA build for a runner with no GPU.

Neither NeuralComposeEEG nor Tests/eval has ever run in CI — build-and-test is
Swift-only. That is how a test importing an uncommitted module reached a pull
request unnoticed, and it left 62 Python tests ungated.

Adds a `python-contracts` job on ubuntu-latest (nothing here is
platform-specific, and the runner is cheaper than macOS).

pytest is a correctness requirement, not a preference. Tests/eval mixes four
harness styles, and most modules are bare `def test_*` functions with no
TestCase and no __main__ runner. `python -m unittest Tests.eval.<module>`
collects ZERO tests from those and prints OK — 8 of 62 tests would have run
behind a green check. pytest collects all 62.

Excluded, each for a stated reason rather than to force green:

  test_joint_embedding.py  imports mlx.core; Apple Silicon only
  3 deselected cases       pre-existing failures on this base, reproducing on
                           both NumPy 1.26.4 and 2.4.6; tracked separately.
                           The rest of their files still run.

Requires the lazy trapezoid resolution in the parent commit: test_session_consume
imports eeg_spectral, which could not import on numpy>=2 before it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@aurascoper

Copy link
Copy Markdown
Owner Author

Tracking issue for the three deselected cases: #38. It records which are wrong tests and which is a real defect — briefly: test_mann_whitney_u_disjoint asserts a p-value that is unattainable at n=3 per group, test_cka_independent ignores biased-CKA inflation when d ≫ n, and cohens_d genuinely returns 0.0 (i.e. "no effect") for two perfectly separated zero-variance groups.

@aurascoper
aurascoper changed the base branch from fix/numpy-trapezoid-compat to docs/eeg-methods-scope July 27, 2026 16:34
@aurascoper
aurascoper merged commit 80d8d8e into docs/eeg-methods-scope Jul 27, 2026
2 checks passed
aurascoper added a commit that referenced this pull request Jul 27, 2026
Brings in #36 (NumPy trapezoid) and #37 (Python contract CI) so this branch is
verified by the non-vacuous job rather than by a local run alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aurascoper added a commit that referenced this pull request Jul 27, 2026
Brings in the structured-state module (#35) that
test_synthetic_structured_state_replay_remains_shadow_only imports, plus the
NumPy fix (#36) and the Python contract CI job (#37).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant