Skip to content

CI: stage eqdata.ITER01/TST-2 so test_equivalence.py actually runs (currently silent SKIP) #192

@k-yoshimi

Description

@k-yoshimi

Summary

`.github/workflows/python-tests.yml` builds `libtrapi.so` for each module but does not invoke `./test_run/run_tests.sh tr_iter01 tr_tst2 ...` to populate `test_run/test_output//eqdata.`. As a result, `python/trlib/tests/test_equivalence.py::TestEquivalence::test_{iter01,tst2}` silently `SKIP`s on CI:

```python

python/trlib/tests/test_equivalence.py:177-184

if not (candidate / knameq).exists():
self.skipTest(
f"eqdata '{knameq}' missing under {candidate}; "
"run `./test_run/run_tests.sh "
f"{fixture_module.BASELINE_NAME}` first."
)
```

This is the `feedback_equivalence_must_pass.md` invisibility pattern called out in MEMORY.md: SKIPped equivalence tests do not count as verification, but CI doesn't surface that they're skipping.

How this manifested as a real bug

PR #187 (2026-05-02 e049a1e L-7b-i) added `AJRFT` to the libtrapi.so C ABI and Python wrapper but missed the paired updates to the Phase-0 dump path (`tr/trregress.f90`, `extract_tr_metrics.py`). On chore branch the local equiv test failed at 1e-10 with `compare_metrics FAIL: scalars.AJRFT: missing` — but CI silently passed for ~10 days because eqdata wasn't staged and the test SKIPped. Discovered by accident during PNBTOT registry work (PR #176 thread, 2026-05-10).

Acceptance

  1. Add a CI step (after the libtrapi.so build loop) that invokes `./test_run/run_tests.sh tr_iter01 tr_tst2` (or whichever subset is feasible — eq builds may need to be installed too)
  2. Also stage similar eqdata for fp/wr/wrx/eq/ti/tot equivalence tests as applicable
  3. Verify the equivalence test row in CI now reports `PASSED` (not `SKIPPED`) for at least `test_iter01` post-fix

Constraints to watch

  • The `run_tests.sh` runner needs eq + tr binaries built. CI currently builds only the *_pic libraries via `make -C libtrapi.so` etc; needs the legacy `make -C eq eq` + `make -C tr tr2` invocations too.
  • Graphics libs (`libg{3d,sp,dp}-gfc64`) are needed for legacy `tr2`/`eq` builds; not currently in the CI workflow's apt install list (`gfortran-12`, `libopenmpi-dev`, etc.). Either install them (preferred) or refactor `tr2` to a graphics-stub variant for CI.

Related

🤖 Filed via post-push retrospective on 2026-05-11.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions