[Spec Change] Formal conformance test suite for runners

[adhabnr-ux]

Flagged in spec/CRITIQUE.md #14 — currently "Partial" in the resolution table, and it's the other item explicitly deferred to v1.1 rather than rushed into 1.0.

The problem: right now, "EvalPort-compliant" is enforced by two things — the JSON Schemas, and each SDK's hand-written validator (validate_suite() / validateSuite() in Python/TypeScript). sdk/python/tests/test_schema_consistency.py and sdk/typescript/tests/schema-consistency.test.ts (added in 1.0.0-rc.1) cross-check that the schemas and the two reference SDKs agree with each other — but there's no independent, spec-owned battery of test documents that a third, non-reference implementation (say, a Rust or Go runner someone builds independently) could run against to claim conformance.

What a conformance suite needs, concretely:

1. A directory of test fixtures — valid documents that MUST pass, invalid documents that MUST fail, with a machine-readable manifest of why each invalid one should fail (which validation rule it violates).
2. Coverage of every numbered rule in Validation Rules, not just the schema shape — e.g. Rule 5 (score clamping to [0,1]) and Rule 6 (score: null vs. scored-failure) both have semantics a JSON Schema alone can't express.
3. Edge cases from the grader type system — the custom type's graceful-skip behavior, the open (non-enum) type string behavior added in 1.0.0-rc.1, the openeval.aggregation strategies (all/any/majority/weighted).
4. A runner-agnostic format for the fixtures themselves (plain JSON in/JSON out, no assumption of pytest or vitest) so it's usable from any language.

Open question for this thread: does this live under spec/conformance/ as spec-owned fixtures (my inclination, since it should be authoritative regardless of implementation language), or as a separate evalport-conformance package that pulls in fixtures and runs them against a runner you point it at? Second one is more useful as an actual CLI tool (evalport-conformance check ./my-runner), but it's more infrastructure to build and maintain.

Good entry point if you want to work on spec tooling rather than a framework adapter.

[adhabnr-ux]

Field data point for whoever picks this up: the 22 adapters shipped so far have all had to pass the real validate_suite()/validate_result_set() validators, and a few of them hit Rule 5 and Rule 6 edge cases worth turning into fixtures rather than reinventing per-adapter.

Rule 5 (score clamping) — real non-[0,1] native scales encountered:

• llamaindex-openeval-adapter: CorrectnessEvaluator scores 1-5 (default score_threshold=4.0). Rescaled as (raw - 1) / 4, original preserved under metadata.openeval.raw_score (well, metadata.llama_index.raw_score in this adapter's own namespace pre-dating that reserved key's landing — worth a conformance fixture that specifically checks a rescaled 1-5 score against the threshold-relative pass/fail, not just the raw number, since 4.0/5 and (4-1)/4 = 0.75 need to agree on passed).
• azure-ai-evaluation-openeval-adapter: several Azure AI evaluators return unbounded or negative-capable continuous scores (similarity metrics) that needed explicit max(0.0, min(1.0, x)) clamping, not just a linear rescale — a good "clamp vs. rescale" distinction for fixtures, since they're not the same operation and picking the wrong one for a given native scale silently distorts results.
• arthur-bench-openeval-adapter: continuous scorers (specificity, bertscore, word_count_match) are already documented [0,1]-native by Arthur Bench itself, but one (hedging_language) is higher-is-worse — not a clamping issue, but a real "don't assume higher score = passed" case that Rule 5's neighbor (the pass/fail-via-threshold language) should probably get a fixture for too.

Rule 6 (score: null vs. scored failure) — this one's more load-bearing than I expected building adapters:

• llamaindex-openeval-adapter hits this directly: EvaluationResult.invalid_result=True (the judge's output couldn't be parsed into a verdict) maps to score: null, passed: false — genuinely distinct from invalid_result=False, score=0.0 (the judge parsed fine and said "no"). Getting this wrong (treating unparseable-judge-output as a scored 0) would silently inflate failure rates whenever a judge model hiccups, which is exactly the kind of bug a conformance fixture should catch and a hand-rolled validator easily misses.
• arthur-bench-openeval-adapter: several built-in scorers are categorical with more than two categories (qa_correctness → incorrect/correct/invalid; summary_quality → reference/candidate/equal/invalid) — the invalid category needs score: null, not folded into the failure category, or a 3-category scorer's failure rate gets silently overcounted by conflating "confidently wrong" with "couldn't judge."

Both look like solid conformance-suite fixtures: "a GraderResult from a real (not synthetic) LLM-judge invalid-parse path must be score: null" and "a 3+-category categorical grader's non-pass/non-fail category must not silently become passed: false with a numeric score." Happy to contribute the actual before/after JSON documents from these two adapters' test suites as a starting fixture set if that's useful groundwork for whichever format (spec/conformance/ vs. a separate CLI package) this thread lands on.

[adhabnr-ux]

Shipped in 1.0.0-rc.3: spec/conformance/ landed with a reference runner (run.py), a README, and 8 fixtures — commit 9634487 — with CI wired in as a dedicated conformance-suite job (commit cbc302a); both green on main as of the latest run.

Went with the first option from the original post — spec-owned fixtures under spec/conformance/, not a separate evalport-conformance CLI package — for the reason you gave: authoritative regardless of implementation language. The CLI-tool idea (evalport-conformance check ./my-runner) is still a reasonable follow-up someone could build against these fixtures, just not a prerequisite for them existing.

Four of the eight fixtures are your field-data-point edge cases verbatim, not hypotheticals:

• null_score_not_scored_failure.json — the llamaindex invalid_result=True case: unparseable judge output must be score: null, passed: false, distinct from a scored 0.
• categorical_grader_invalid_category.json — the arthur-bench 3-category case (incorrect/correct/invalid), same Rule 6 distinction reached from a closed category enum instead of free text.
• score_out_of_range_rejected.json and boolean_score_rejected.json — Rule 5's clamped range, plus the score-must-never-be-boolean cross-language gotcha (Python's bool is an int subclass).

The other four cover the custom/non-standard grader type contract (custom_grader_missing_handler_rejected.json, non_standard_grader_type_with_handler_valid.json) and the two conventions that landed alongside this in the same rc (partial_resultset_resumable_run.json for #10, judge_hardening_self_report.json for #11).

Format matches your point 4 — plain {type, expect, document} JSON, no pytest/vitest dependency, so a Rust or Go implementation can load fixtures/*.json directly and check its own validator's answer against expect.valid with zero dependency on this repo's code. run.py is this repo's own self-check against the real hand-rolled validator; going a bit further than the original ask, every fixture is also independently checked against the raw JSON Schema files (Draft202012Validator), so expect.valid reflects agreement between both validation paths this repo maintains, not just one.

Not claiming this is exhaustive — the README's "What this doesn't cover" section says so explicitly: no CLI runtime-behavior fixtures yet (cost estimation, retry logic, grader execution), no referential-integrity fixtures beyond what validate_suite() already checks. Real extension point for whoever wants it next.

Marking this thread's core ask resolved. Thanks for the concrete Rule 5/6 examples above — they're the reason this suite shipped with real fixtures instead of synthetic placeholders.