Skip to content

test: trim tautological self-checks from participant spec-oracle suites #558

Description

@Brad-Edwards

Problem

A behavioural audit of the Python test suites — checking for the "assert against yourself / mock the thing under test" anti-pattern that affected a sibling project — found that two executable spec-oracle test files contained acceptance-direction tests that are tautological:

  • implementations/python/tests/test_participant_semantics_invariant_oracle.py
  • implementations/python/tests/test_participant_runtime_invariants.py

Both files re-encode a formal spec's invariants as test-local predicates with no production callers (confirmed: zero references under src/ or packages/). Several tests asserted that a co-authored fixture / Hypothesis generator — constructed to satisfy those same predicates — does in fact satisfy them. Those checks pass by construction, cannot catch a wrong implementation, and inflate the apparent coverage of participant-semantics / participant-runtime behaviour when no production code is exercised.

The rest of the audit was clean: real parser/validator/runtime/conformance/CLI/MCP code runs across the suite. This issue is scoped to the two oracle files.

Fix

  • Remove the tautological acceptance-direction tests:
    • semantics oracle: the catalog self-check and both Hypothesis "generator output satisfies the predicates" tests, plus the now-dead generator strategy and its imports.
    • runtime oracle: the six bare *_accepts_generated_valid_traces tests.
  • Keep the load-bearing tests: the spec to catalog drift guard, the per-invariant mutation/rejection differential tests, the *_accepts_supported_*_variants over-rejection guards, and the canonical positive control (now commented as such).
  • Relabel both module docstrings to state plainly that these are spec self-consistency oracles, not tests of production code, and to point at the suites that cover runtime enforcement behaviourally (test_sem_211_*test_sem_218_*, test_run_305/306/311_*).

Verification

  • ruff check clean; affected suites pass (35 tests); full suite still collects (2523 tests).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions