Proposal: EvalPort adapter for ifixai.api.run_inspections() / TestRunResult

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange spec + SDK (evalport-sdk on PyPI, openeval.validate.validate_result_set()) for portable LLM eval results, with ~35 framework adapters so far. I read ifixai/core/types.py, ifixai/harness/base.py, and docs/python-api.md and think iFixAi's result model is one of the cleanest fits I've seen for this.

Why it fits: ifixai.api.run_inspections(...) (async) returns a TestRunResult (pydantic) with test_results: list[TestResult], overall_score, grade, passed, category_scores. Each TestResult carries test_id, score, threshold, passed, status (TestStatus), and evidence: list[EvidenceItem], where EvidenceItem has prompt_sent, expected/expected_behavior, actual/actual_response, passed, and (for judge-scored items) judge_verdict/dimension_scores/rubric_weighted_score. That's already a per-test-case input/expected/actual/pass-fail record — exactly what OpenEval's ResultSet wants, at the EvidenceItem level, rolled up per inspection at the TestResult level.

Sketch (before → after):

# before: ifixai-native
import asyncio
from ifixai.api import run_inspections

result = asyncio.run(run_inspections(
    provider="openai", api_key="sk-...", model="gpt-4o",
    fixture="default", system_name="my-agent",
))
print(result.overall_score, result.grade)
# result: TestRunResult(test_results=[TestResult(test_id, score, threshold,
#          passed, status, evidence=[EvidenceItem(prompt_sent, expected,
#          actual, passed, judge_verdict, ...)])])

# after: ifixai-openeval-adapter
from ifixai_openeval_adapter import results_to_openeval
from openeval.validate import validate_result_set

result_set = results_to_openeval(result)   # TestRunResult -> OpenEval ResultSet
validate_result_set(result_set)             # portable, comparable across tools

The adapter would map each EvidenceItem to an OpenEval result row (test_case_id → id, prompt_sent → input, expected_behavior → expected, actual_response → actual, passed/rubric_weighted_score → score+verdict), and each TestResult's test_id/category/threshold/status (including INCONCLUSIVE/ERROR, not just pass/fail) into the ResultSet's per-suite grouping, with TestRunResult.overall_score/grade as the summary.

I'd like to build adapters/ifixai-openeval-adapter/ (standard layout — pyproject.toml with a self-referencing pinned extra per the EvalPort convention, src/ifixai_openeval_adapter/__init__.py exposing to_openeval()/results_to_openeval(), tests/test_adapter.py, README.md) and submit it as a PR, either here or under EvalPort's adapters/ directory — whichever you'd rather host it in.

One question before I start: since status includes INCONCLUSIVE and ERROR alongside PASS/FAIL, would you want those mapped to a skipped/error OpenEval verdict, or excluded from the portable set entirely (matching how insufficient_evidence already excludes them from your own scoring denominator)? Happy to follow whichever convention you'd prefer.

— Sahi (independent contributor, not affiliated with iFixAi)

[n-papaioannou]

Hey Sahi, thank you for reading our codebase so closely. The mapping you sketched seems right.
Please host it under EvalPort's adapters/ directory as recommended. That keeps our repo focused on the engine, and you can evolve the adapter with your spec without waiting on us.
Regarding the statuses, map INCONCLUSIVE and ERROR to your skipped/error verdicts rather than dropping them. We exclude them from the scoring denominator but never hide them, and the portable set should show the same picture.
Tag us on the PR so that we can take a look.

[adhabnr-ux]

Thanks, Nikos — good catch on the statuses. I'll map INCONCLUSIVE and ERROR to skipped/error verdicts respectively rather than dropping them, and keep them excluded from the scoring denominator the same way you do internally, so the portable set shows the same picture instead of implying something the original run didn't.

I'll host it under EvalPort's adapters/ as planned and tag you both here and on the PR once it's up.