test(trlib): mirror eqlib eqdata fallback to fix silent SKIP (#192)#195
Merged
Conversation
Approach A2 (eq-mirror fallback) replaces the original A1 (CI step fixture-copy). Pivot driven by Codex spec review #1 finding: eq's test_equivalence.py:197-205 ALREADY has the FIXTURES_DIR fallback; trlib is missing it. Adding the same fallback to trlib is the root- cause fix vs adding a CI shim. Scope (3 files, 1 commit): - python/trlib/tests/test_equivalence.py: add FIXTURES_DIR constant + replace SKIP-only logic with two-tier fallback mirroring eq:53, 197-211 verbatim. - python/trlib/tests/fixtures/eqdata.ITER01 (NEW, 45956 B): copy of python/eqlib/tests/fixtures/eqdata.ITER01 (eqdata.TST-2 mirror is already present in trlib/tests/fixtures/). - .gitignore: add !python/trlib/tests/fixtures/eqdata.ITER01 negation next to the existing TST-2 negation. Out of scope (follow-ups): - fp/wr/wrx/ti silent-SKIP audit (different mechanism, no KNAMEQ) - eq.x build for CI regen / eq-physics drift detection Drift risk (Codex MED): CI gfortran-13.2 vs baseline's 13.3 may produce > 1e-10 drift; §6.4 documents the contingency regen path. Reviewer trail: brainstorming → Codex spec review #1 → pivot A1→A2 → this spec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bite-sized task plan for the spec at docs/superpowers/specs/2026-05-12-ci-tr-equiv-staging-design.md (committed be5245e). 8 numbered tasks across 2 phases: - Pre-flight (worktree + branch + current-SKIP-state baseline) - Phase 1 (5 tasks): copy eqdata.ITER01 fixture, .gitignore negation, FIXTURES_DIR + 2-tier fallback in test_equivalence.py, local sanity, commit C1 - Phase 2 (3 tasks): bounded pytest + parallel reviewers (in-house + Codex) on diff + REVIEW_OK marker + push + PR create + Bugbot trigger Each task has exact files/lines, before/after snippets, verification commands, expected outputs. Acceptance maps to issue #192 #1-#3. Self-review: spec coverage complete, no placeholders, names consistent across tasks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… runs (#192) Mirror python/eqlib/tests/test_equivalence.py's two-tier eqdata fallback into python/trlib/tests/test_equivalence.py so TestEquivalence::test_{iter01,tst2} run on CI (and on any fresh checkout) instead of silently SKIPping. Closes the invisibility gap that let PR #187 (L-7b-i AJRFT) pass CI for ~10 days with a real 1e-10 failure under chore branch's local test. Three changes: - python/trlib/tests/test_equivalence.py: add FIXTURES_DIR constant (line 46 area, between TEST_OUTPUT_DIR and COMPARE_SCRIPT) + replace SKIP-only logic in _check_case with the two-tier fallback (prefer Phase-0 runner output at TEST_OUTPUT_DIR/<case>/<KNAMEQ>, fall back to FIXTURES_DIR/<KNAMEQ>, only SKIP if both absent). Verbatim structural mirror of eqlib test_equivalence:53, 197-211. - python/trlib/tests/fixtures/eqdata.ITER01 (new, 45956 B): exact copy of python/eqlib/tests/fixtures/eqdata.ITER01. The TST-2 mirror already lives at python/trlib/tests/fixtures/eqdata.TST-2. - .gitignore: add !python/trlib/tests/fixtures/eqdata.ITER01 negation, adjacent to the existing TST-2 negation. Out of scope (follow-up issues to be filed): - fp/wr/wrx/ti silent-SKIP audit (different mechanism, no KNAMEQ). - CI-side eq.x build for fresh eqdata regen (eq-physics drift detection); the current fixture-trust model mirrors the project's existing convention. Spec: docs/superpowers/specs/2026-05-12-ci-tr-equiv-staging-design.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit e2182bf. Configure here.
CI surfaced what #192's eq-mirror fallback was designed to expose: test_tst2 now actively runs (no longer silent SKIP) and fails with `scalars.AJRFT: missing` because `test_run/baselines/tr_tst2/ metrics.json` predates PR #187 (AJRFT addition). #190 tracks the baseline regen, which is in turn blocked by upstream eq_tst2 drift (~3e-9 > 1e-10). Until that chain resolves, mark test_tst2 as xfail(strict=True) so: - CI is green again (this PR's #192 goal preserved for tr_iter01) - When #190 closes and the baseline is regenerated to include AJRFT, the xfail flips to XPASS and CI fails red, forcing removal of this decorator (CLAUDE.md narrow exception protocol). This is the documented contingency path from spec §6.4, except the cause is not gfortran drift — it's a pre-existing baseline staleness that was hidden by the silent SKIP this PR fixed. In a real sense #192 is now doing its job: invisibility → visibility. test_iter01 still PASSes (its baseline was regenerated in 24b1b12 to include AJRFT). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 43afff1. Configure here.
This was referenced May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the silent-SKIP invisibility class on
python/trlib/tests/test_equivalence.py::TestEquivalence::test_{iter01,tst2}by adding a two-tier eqdata fallback that mirrorspython/eqlib/tests/test_equivalence.py:53, :197-211. Closes #192.Why fallback rather than CI shim
Original issue #192 proposed a CI workflow step pre-staging eqdata. Codex spec review (round 1) surfaced that
python/eqlib/tests/test_equivalence.py:197-211ALREADY has the two-tier fallback (preferTEST_OUTPUT_DIR, fall back toFIXTURES_DIR).trlibwas the asymmetric module. Mirroring eq's pattern is the root-cause fix: it works on CI and on any fresh checkout, with no CI yaml change required.Changes (3 files, 1 commit)
python/trlib/tests/test_equivalence.pyFIXTURES_DIRconstant (line 46) + replace SKIP-only block with 2-tier fallback (lines 179-195). Verbatim structural mirror of eq's:53, :197-211python/trlib/tests/fixtures/eqdata.ITER01python/eqlib/tests/fixtures/eqdata.ITER01.eqdata.TST-2mirror already exists in the same dir.gitignore!python/trlib/tests/fixtures/eqdata.ITER01negation, adjacent to existing TST-2 negation (per-module grouping pattern)No CI workflow change. No Fortran build. No other source touched.
Verification
pytest python/trlib/tests/test_equivalence.py -v→ bothtest_iter01andtest_tst2SKIPPED at outer@unittest.skipUnless(DEFAULT_SO.exists())(libtrapi.so not built on mac). The fallback IS in place; CI will exercise the path end-to-end.pytest python/eqlib/tests/test_equivalence.py --collect-onlyunchanged (2 tests collected, not modified).pytest python/totlib/ python/trlib/ python/eqlib/ test_run/scripts/tests/ --forked --timeout=120): 236 passed, 92 skipped, 2 pre-existing failures.test_iter01andtest_tst2should reportPASSED, notSKIPPED. Inspect Python 3.11 and 3.13 matrix logs.Drift contingency (spec §6.4)
If CI fails at 1e-10 (CI gfortran-13.2 vs baseline-host gfortran-13.3 ULP drift), regen
test_run/baselines/tr_{iter01,tst2}/metrics.jsonon Ubuntu gfortran-13.2 and add as follow-up commit. Both reviewers (in-house + Codex) concur this is< 20%probability based on prior cross-host runs.Pre-existing failures (NOT introduced)
test_close_raises_exception_group_when_multiple_modules_fail— Python 3.10 lacksExceptionGroup(test docstring notes; CI runs 3.11+, this test PASSes there).test_fails_when_baseline_is_tiny_and_actual_is_zero— pre-existing ondevelop(compare_metricsnumeric edge case at 1e-300 vs 0).git diff origin/develop --name-only -- python/totlib/tests/test_pipeline.py test_run/scripts/tests/test_compare_metrics.pyis empty — neither test file is touched.Follow-up issues to file after PR opens
CI: investigate fp/wr/wrx/ti silent SKIPs in test_equivalence— different mechanism (noKNAMEQ), separate audit.CI: build eq.x graphics-stubbed to regen eqdata.* for physics-drift detection— long-term improvement.Review trail
docs/superpowers/specs/2026-05-12-ci-tr-equiv-staging-design.md(commitbe5245e7).docs/superpowers/plans/2026-05-12-ci-tr-equiv-staging.md(commit148a27ca).🤖 Generated with Claude Code
Note
Medium Risk
Changes test execution semantics by making
trlibequivalence tests run (or xfail) on CI instead of skipping, and adds a new committed binary fixture; this may surface baseline drift or hide a real failure behind the new strictxfailontest_tst2.Overview
Ensures
python/trlib/tests/test_equivalence.pyno longer silently SKIPs wheneqdata.*is missing by addingFIXTURES_DIRand a two-tier lookup: prefertest_run/test_output/<case>/but fall back to committed fixtures inpython/trlib/tests/fixtures/.Adds a new tracked binary fixture
python/trlib/tests/fixtures/eqdata.ITER01(and updates.gitignoreto allow it), and marksTestEquivalence.test_tst2as a strictpytestxfailwith an issue reference until the baseline is regenerated.Reviewed by Cursor Bugbot for commit 43afff1. Bugbot is set up for automated code reviews on this repo. Configure here.