EVA ↔ EvalPort adapter — portable scenario/metric interop?

[adhabnr-ux]

Hi — I maintain EvalPort, an open interchange spec for portable LLM eval data (TestCase/Grader/Result/ResultSet/GraderResult, with a Python SDK — pip install evalport-sdk, import openeval — and a TS one). I read through src/eva/models/record.py, results.py, and metrics/base.py and think there's a fairly mechanical mapping worth discussing before I write any code, per CONTRIBUTING.md's guidance to raise larger changes here first.

The mapping, as I read it:

EVA	EvalPort
EvaluationRecord (id, user_goal, scenario_context, ground_truth.expected_scenario_db, category)	TestCase (id, input, expected_output, metadata)
MetricScore (name, score, normalized_score, error, skipped)	GraderResult (grader_id, score, passed, reason)
RecordMetrics (record_id, metrics: dict[str, MetricScore])	Result (test_case_id, grader_results)
a run's aggregate_metrics (EVA-A/EVA-X/EVA-overall)	ResultSet.summary

BaseMetric.metric_type maps reasonably onto EvalPort's grader type enum for two of three cases, and genuinely doesn't for the third:

• TEXT_JUDGE (e.g. faithfulness) → llm_judge, clean fit.
• AUDIO_JUDGE (speech_fidelity, scoring the mixed audio directly) → no analog. EvalPort's grader types are all text/structure graders; there's no "grades raw audio" type. I'd rather flag that as an open spec question than silently drop those scores on export.
• CODE (task_completion) → doesn't cleanly fit EvalPort's code type either, which the spec describes as a sandboxed custom function. task_completion is actually a deterministic hash/diff comparison against expected_scenario_db — closer to a structured-state equality check than executable code. Might map better as json_path/json_schema against a serialized DB state, or need its own semantics.

Concrete sketch, following the to_openeval()/from_openeval() shape used by adapters/autogen-openeval-adapter — the pattern EvalPort's README points to as the reference shape to copy:

# eva_openeval_adapter/__init__.py
from openeval.types import OPENEVAL_VERSION

def record_to_testcase(record: "EvaluationRecord") -> dict:
    return {
        "id": record.id,
        "input": record.user_goal,
        "graders": ["gr_task_completion", "gr_faithfulness"],  # per active MetricRegistry
        "metadata": {"scenario_context": record.scenario_context, "category": record.category},
        "expected_output": record.ground_truth.expected_scenario_db,
    }

def metrics_to_result(rm: "RecordMetrics") -> dict:
    grader_results = [
        {
            "grader_id": name,
            "score": ms.normalized_score,
            "passed": bool(ms.normalized_score and ms.normalized_score >= 0.5),
            "reason": ms.details.get("explanation"),
        }
        for name, ms in rm.metrics.items() if not ms.skipped
    ]
    return {
        "test_case_id": rm.record_id,
        "passed": all(g["passed"] for g in grader_results),
        "grader_results": grader_results,
        "metadata": {"aggregate_metrics": rm.aggregate_metrics},
    }

Nothing here touches EVA internals — it'd be a standalone package like the AutoGen one, validated against openeval.validate.validate_suite()/validate_result_set().

Two questions before I'd invest time in a PR:

1. Is scenario/result portability something EVA wants, given it's a fairly specialized voice-agent harness rather than a general text-eval framework?
2. Independent of the adapter — is audio-native judging (raw-audio-in grading, not just audio-derived text) worth an EvalPort spec issue regardless? EVA is the first real case I've seen that needs it.

— Sahi, independent contributor (not affiliated with this project)