Interop idea: a unitxt-openeval-adapter for portable eval results

[adhabnr-ux]

EvalPort (https://github.com/adhabnr-ux/evalport) is a small open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM evaluation datasets and results — a JSON schema plus openeval.validate.validate_suite() / validate_result_set(), so results produced by one eval framework can be read, diffed, or re-checked by another tool without re-running anything. It's adapter-based: 34 frameworks already have a thin, zero-footprint adapters/<name>-openeval-adapter/ package that just converts to/from the interchange shape.

Why unitxt fits well: evaluate() in unitxt.api (as in your own README example) returns an EvaluationResults — a list of per-instance dicts, each carrying task_data, prediction/processed_prediction, references/processed_references, and a score dict split into instance and global. .instance_scores flattens each instance into task_data + prediction + per-instance score fields, and .global_scores gives the aggregate GlobalScores (score, score_name, num_of_instances, plus CI bounds when calc_confidence_intervals=True). That's already close to EvalPort's shape — a list of per-instance results plus one aggregate summary — it just isn't in the interchange format.

Sketch, using the real API:

from unitxt import evaluate, create_dataset
# ... build dataset, get predictions, as in the unitxt README example ...
results = evaluate(predictions=predictions, data=dataset)

# proposed: unitxt_openeval_adapter
from unitxt_openeval_adapter import results_to_openeval
result_set = results_to_openeval(results)  # walks results.instance_scores + results.global_scores

from openeval.validate import validate_result_set
validate_result_set(result_set)  # now portable / framework-agnostic

Honest caveat: task_data is arbitrary per task, and vision/speech tasks stuff non-text payloads into it. I'd target unitxt's very common text-in/text-out tasks first and treat other modalities' task_data as opaque metadata rather than claim full fidelity there.

If this is of interest, I'm happy to build adapters/unitxt-openeval-adapter/ — zero footprint on unitxt itself, following the pyproject convention already established across the other 34 adapters (self-referencing pinned unitxt extra + test extra, per evalport Discussion #13) — and submit it as a PR to evalport, referencing this discussion.

— Sahi, independent contributor (not affiliated with IBM or the unitxt team)