EvalPort update: merged into Inspect AI, 20 framework adapters shipped, spec governance is now live

[adhabnr-ux]

A few weeks in, here's where EvalPort actually stands — with links so you can check every claim yourself rather than take my word for it.

Merged into a real framework

UK AISI's Inspect AI merged EvalPort into their official community extensions list on August 11. That's the first time EvalPort has landed inside another project's own repository rather than living as a standalone adapter package — genuinely useful validation that the interchange format holds up outside our own docs.

20 framework adapters, all real and tested

Every adapter under adapters/ is an installable Python package with a real pyproject.toml, to_openeval()/from_openeval(), and tests that run against the actual framework's SDK (not mocks) plus the real EvalPort validator: Argilla, AutoGen, Braintrust, CrewAI, DSPy, Evidently, Giskard, Guardrails, Haystack, Langfuse, LangSmith, LlamaIndex, MLflow, Opik, Patronus, Phoenix, Ragas, uptrain, Vertex AI, and Weave.

A few of these came from outside contributors, not just from this project — including a community-authored PR on openai/openai-python adding OpenEvalItem import/export helpers.

Spec governance is now real, not aspirational

spec/SPEC.md has a proper Governance section now — how decisions get made, what it takes to become a collaborator. If you want to shape the spec itself rather than ship a framework adapter, there are four open RFC discussions with no prior contribution required:

• #8 — Suite/result signing for integrity verification
• #9 — Formal conformance test suite for runners
• #10 — Resuming interrupted runs and merging partial ResultSets
• #11 — Should llm_judge prompt-injection mitigations be a MUST, not a SHOULD?

What's still genuinely unfinished (not hiding this)

pip install evalport-sdk / npm install evalport-sdk currently gets you 1.0.0, not the 1.1.0 that's tagged as a release and documented in the README (Benchmark Hub, evalport run CLI). The CI publish jobs were failing because of a token-configuration gap — fixed the root cause by switching to OIDC Trusted Publishing (no more long-lived tokens in the repo at all), tracked in #7. One small registry-side step is still outstanding before 1.1.0 actually ships — should be resolved soon.

Also open and waiting on maintainer review: truera/trulens#2697 and microsoft/autogen#8009.

If you want to get involved

The RFC discussions above are the easiest entry point if you have an opinion on the spec. If you maintain or use an eval framework not in the adapter list, issue #6 tracks "adapter wanted" requests — or just open one. Everything's Apache 2.0.