packages/agents Evals data model → EvalPort (portable eval result JSON) — worth a reporter?

[adhabnr-ux]

Hi maintainers,

I'm Sahi, maintainer of EvalPort — an Apache-2.0 open interchange spec for portable LLM evaluation TestCase/Grader/ResultSet data, with SDKs in TypeScript and Python (evalport-sdk) and 32 tested framework adapters (Ragas, Braintrust, MLflow, LangSmith, DeepEval, etc. — see adapters/).

I went through packages/agents/src/Evals/ before writing this — it's a genuinely rich, well-designed eval model: EvalResult (id(), description(), verdict() returning EvalVerdict::Passed|Failed|Scored|Skipped, assertions(), duration(), error(), tokens()), AssertionResult (name(), score() 0–1, severity(), threshold(), passed(), message()), JudgeScore (score, reason, evidence, optional run), EvalDataset/EvalDatasetRow, and EvalRunResult (aggregate counts/tokens()/provenance()). It's one of the more complete eval result shapes I've come across in any language, and it maps cleanly onto EvalPort's TestCase/Grader/ResultSet JSON shapes.

Since this is a PHP project and EvalPort's own SDKs are TS/Python, I'm not proposing a package dependency — just flagging that the two shapes line up well enough that a small, dependency-free serializer living entirely inside instructor-php (in the spirit of the existing JUnitEvalReporter/ArtifactEvalReporter, implementing CanReportAgentEvals) could let CI-produced eval results interop with other eval tooling via a shared JSON format — the same role ragas-openeval-adapter and braintrust-openeval-adapter play for their ecosystems in Python.

Concretely, the mapping (real fields on both sides):

• EvalResult::id() / verdict() / duration() / error() → a ResultSet result: {test_case_id, passed: verdict() === Passed, duration_ms: duration()*1000, error}
• AssertionResult::name() / score() / passed() / message() → that result's grader_results[] entry: {grader_id: name(), type: 'custom', score: score(), passed: passed(), reason: message()}
• EvalDatasetRow::toArray() → an EvalPort TestCase (input, expected_output, graders, metadata) almost as-is
• EvalRunResult::toArray()'s counts/tokens()/provenance() → ResultSet.summary / ResultSet.metadata

A small illustrative sketch, using only the real accessors above:

function toEvalPortResult(EvalResult $r): array {
    return [
        'test_case_id' => $r->id(),
        'passed' => $r->verdict() === EvalVerdict::Passed,
        'duration_ms' => (int) round($r->duration() * 1000),
        'grader_results' => array_map(
            fn (AssertionResult $a): array => [
                'grader_id' => $a->name(),
                'type' => 'custom',
                'score' => $a->score(),
                'passed' => $a->passed(),
                'reason' => $a->message(),
            ],
            $r->assertions()->all(),
        ),
    ];
}

This is a discussion, not a PR — I wanted to check whether this is something you'd even want before writing one (e.g. as an optional EvalPortReporter alongside the existing reporters). Happy to draft it if there's interest, or this can just sit here as food for thought.

Repo: https://github.com/adhabnr-ux/evalport
Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md

— Sahi, independent contributor (not affiliated with this project)