Portable eval results: an EvalPort adapter for AgentLab's GAIA/OSWorld tasks + ExpResult traces

[adhabnr-ux]

Hi — I'm Sahi, an independent contributor working on EvalPort, a small open interchange format for LLM/agent eval data (TestCase/Grader in, Result/ResultSet/GraderResult out — plain dataclasses + JSON Schema, Python + TS SDKs). Posting here rather than opening a PR because this is a "does the mapping make sense" question before I write any code against your internals.

Not affiliated with ServiceNow/AgentLab — just think the fit is close enough to be worth a converter.

Why AgentLab specifically: unlike most agent frameworks, AgentLab already produces a lazily-loaded, tabular result format (load_result_df / ExpResult.get_exp_record() in analyze/inspect_results.py) and a full step-level trace format (the Tape/tape.json model in analyze/tapes.py). That's unusually close to EvalPort's own split between a ResultSet (run-level rollup) and per-Result grader_results — most of the adapters I've written so far have to synthesize one of those two levels from nothing.

Concrete mapping I'd propose

Input side — GAIA task → TestCase (using the real field names from benchmarks/gaia.py):

def gaia_task_to_testcase(task: dict) -> TestCase:
    return TestCase(
        id=task["task_id"],
        input=task["Question"],
        expected_output=str(task["Final answer"]),
        graders=["gr_gaia_scorer"],
        metadata={"level": task["Level"], "file_path": task.get("file_path")},
    )

# question_scorer() already IS a grader — it just needs a Grader wrapper,
# type="custom" since it does numeric/list/string normalization, not a plain
# exact_match:
gr_gaia_scorer = Grader(
    id="gr_gaia_scorer",
    type="custom",
    description="GAIA numeric/list/string-normalized answer match (question_scorer)",
    params={"handler": "agentlab.benchmarks.gaia:question_scorer"},
)

Output side — ExpResult → Result/GraderResult, one ResultSet per study:

def exp_result_to_result(exp_result) -> Result:
    rec = exp_result.get_exp_record()
    reward = rec.get("cum_reward")
    passed = bool(reward) and reward >= 1.0
    return Result(
        test_case_id=rec["env.task_name"],   # e.g. "gaia.<task_id>"
        passed=passed,
        grader_results=[GraderResult(
            grader_id="gr_gaia_scorer",
            type="custom",
            score=reward,
            passed=passed,
            reason=rec.get("stack_trace") if rec.get("err_msg") else None,
        )],
        duration_ms=None,  # AgentLab doesn't track this per-episode today; open Q below
        error={"message": rec["err_msg"]} if rec.get("err_msg") else None,
        metadata={"agent": rec.get("agent.agent_name"), "n_steps": rec.get("n_steps")},
    )

def study_to_resultset(exp_dir, suite_id="gaia") -> ResultSet:
    from agentlab.analyze.inspect_results import load_result_df
    df = load_result_df(exp_dir, set_index=False)
    results = [exp_result_to_result(r) for r in df["exp_result"]]
    return ResultSet(
        version=OPENEVAL_VERSION,
        suite_id=suite_id,
        run_id=Path(exp_dir).name,
        started_at=...,  # from Study reproducibility metadata
        results=results,
        runner={"name": "agentlab"},
    )

For the trace side, tapes.py's Tape.metadata.reward/.task/.error and the step-level kind (*_action/*_thought/*_observation) map cleanly into Result.metadata / GraderResult.metadata if you want step-level detail preserved rather than collapsed to a single pass/fail — happy to sketch that direction too if there's interest, since it's what agent_xray/episode_to_html already render.

Open questions before I'd actually build this:

1. Is cum_reward >= 1.0 the right passed threshold across all benchmarks, or does e.g. WorkArena L2/L3 use partial credit where that's wrong?
2. Would you rather this live as a standalone agentlab-openeval-adapter package (same pattern as the CrewAI adapter — works against your public ExpResult/Tape shapes from the outside, no changes needed on your side), or is there appetite for a thin agentlab.analyze.openeval_export module in-tree? I lean toward standalone given the [WARNING] in your README about scope, but wanted to ask rather than assume.

Genuinely just want to know if this is useful before spending time on it — GAIA + OSWorld would be my first two targets since they already have clean scalar rewards.

— Sahi, independent contributor (not affiliated with this project)