EvalPort adapter for LangTest — exporting Harness.generated_results as a portable ResultSet

[adhabnr-ux]

EvalPort (https://github.com/adhabnr-ux/evalport) is an open interchange spec + small SDK (evalport-sdk on PyPI) for portable LLM evaluation datasets and results — a shared EvalSuite/ResultSet JSON shape so results from one eval framework can be validated, diffed, or replayed by another instead of being locked into that framework's own report format.

LangTest's own result model fits this well because it's already framework-agnostic on purpose — the README says as much: "one only needs to call access the is_pass property to assess whether the expected_results and the actual_results are the same, regardless [of] the downstream task." Concretely, after

h = Harness(task='ner', model={"model": 'dslim/bert-base-NER', "hub": 'huggingface'})
h.generate().run().report()

h.generated_results is a flat List[Sample], and every sample — NER, QA, robustness, bias, whatever task — is a BaseSample (langtest/utils/custom_types/sample.py) carrying original, test_type, test_case, category, expected_results, actual_results, and an is_pass() check. langtest/utils/report_utils.py::model_report() then reduces that list into the DataFrame h.report() prints, grouped by test_type/category with pass_count/fail_count/pass_rate/minimum_pass_rate/pass columns. That's precisely a per-test-case pass/fail-with-scoring result set, computed by exactly one code path — a much easier adapter target than frameworks that only expose a rendered report.

A langtest-openeval-adapter (following the standalone-package, zero-footprint pattern used for the other 34 adapters in EvalPort — see Discussion #13 for the pyproject convention) would do:

from langtest_openeval_adapter import results_to_openeval

h = Harness(task="ner", model={"model": "dslim/bert-base-NER", "hub": "huggingface"})
h.generate().run().report()

result_set = results_to_openeval(h.generated_results, suite_id="ner_typo_suite", run_id="run_1")
# -> {"results": [{"test_case_id": ..., "actual_output": ..., "passed": sample.is_pass(),
#                   "grader_results": [{"grader_id": sample.test_type, "passed": ..., "metadata": {"category": sample.category}}]}],
#     "summary": {"total": ..., "passed": ..., "pass_rate": ...}, ...}

from openeval.validate import validate_result_set
validate_result_set(result_set)  # confirms EvalPort compliance

So a bias/robustness/accuracy run from LangTest becomes a portable ResultSet any other EvalPort-compliant tool can read, diff across runs, or re-score — without touching anything in this repo.

I'd like to build this and open it as a PR against adapters/ in the EvalPort repo (not against langtest) if that's welcome — happy to adjust the mapping if you think a different set of BaseSample fields belongs in the exported result instead.

— Sahi, independent contributor, maintainer of EvalPort

[chakravarthik27]

Thanks, @adhabnr-ux.

This looks like a good fit for LangTest, and I agree that BaseSample is the right place to build the adapter from.

Please go ahead with the PR under adapters/. The proposed mapping looks good to me. We can review and adjust the field mapping if needed once the PR is up.

Thanks for contributing!

[adhabnr-ux]

Thanks for the quick yes — I'll build langtest-openeval-adapter as a standalone package under EvalPort's adapters/, following the mapping as proposed: results_to_openeval() over Harness.generated_results, one Result per Sample with passed from is_pass(), and test_type/category carried through on the grader result. I'll open the PR against adapters/ once it's built and tested against the real langtest package (not mocks), and tag you here for the field-mapping review as you offered. Thanks for engaging with this directly rather than leaving it sit.

— Sahi