Add OpenEval adapter for AutoGen evals

[Dresden (DresdenGman)]

Summary

Add a small OpenEval adapter in autogenstudio/eval so AutoGen eval data can be exported to and imported from the OpenEval format.

Why

Issue #8005 proposed native import/export support. The maintainer guidance pointed to a new module boundary and a minimal to_openeval() / from_openeval() surface, so this keeps the integration narrow and localized.

What changed

• Added autogenstudio.eval.openeval with to_openeval() and from_openeval() helpers.
• Re-exported the adapter from autogenstudio.eval.
• Added the renamed evalport-sdk distribution as a runtime dependency while retaining the backward-compatible openeval Python imports.
• Updated the workspace lockfile for evalport-sdk 1.0.0.
• Added a regression test covering a round-trip between AutoGen tasks and OpenEval payloads.

Validation

• uv lock --check.
• Verified evalport-sdk==1.0.0 provides openeval.types and openeval.validate.
• python3 -m py_compile on the new module and test file.
• Dependency-free smoke test that loaded the module with stubs and verified the conversion behavior.

References

Fixes #8005.

[Dresden (DresdenGman)]

@microsoft-github-policy-service agree

[adhabnr-ux]

Thanks for this PR, Dresden! I'm the OpenEval spec author — this looks great. A few observations:

What works well:

• Clean module boundary in autogenstudio/eval/openeval.py
• to_openeval() / from_openeval() surface is exactly what we proposed
• Round-trip test is solid — covers single and multi-input tasks, expected_tools, empty outputs
• Graceful ImportError fallback when SDK isn't installed
• expected_tools mapping from metadata is a nice touch — that's exactly how our agent eval profile works

Minor suggestions (non-blocking):

1. The gr_output_match grader is hardcoded to exact_match. For agent evals, you might want to also support llm_judge as an option — users could configure which grader type to use via metadata. But this is fine for v1.
2. The expected_outputs field (plural) in the test case payload isn't in the OpenEval spec — only expected_output (singular) is standard. The extra field is harmless (unknown fields are ignored per our forward-compatibility rules), but if you want strict compliance, you could drop it.
3. OPENEVAL_VERSION fallback to "unknown" — the spec requires semver format. If the SDK isn't installed, consider defaulting to "1.0.0" instead.

Overall: This is exactly the kind of integration we hoped for. The code is clean, tested, and follows the spec correctly. Happy to see openeval-sdk as a runtime dependency.

If the AutoGen maintainers want to wait for wider adoption before merging (as Charles Teague (@dragonstyle) mentioned on the Inspect AI issue), this PR could also live as a standalone package (autogen-openeval-adapter) on PyPI. Either way works.

Great work! 🎉

[adhabnr-ux]

Thanks Dresden (@DresdenGman) — confirmed the update looks correct. The dependency is now evalport-sdk in pyproject.toml, imports stay under openeval.* (which is the backward-compatible Python import name), and uv.lock is regenerated.

From the spec author's perspective, this is ready for maintainer review. The adapter correctly maps:

• AutoGen EvalTask → EvalPort TestCase (id, input, expected_outputs, expected_tools)
• EvalPort suite → AutoGen EvalTask list via from_openeval()
• Grader: exact_match with ignore_case (reasonable default)

No blocking issues from my side.

[adhabnr-ux]

Friendly nudge — the dependency has been updated to evalport-sdk and the PR was reviewed by the spec author (see review). Is there anything else needed before this can be reviewed by a maintainer? Happy to make any adjustments.

[Dresden (DresdenGman)]

Ready for maintainer review. The dependency has been updated to evalport-sdk, the openeval.* imports remain for backward compatibility, and the lockfile has been regenerated.

[Dresden (DresdenGman)]

No further changes from my side at the moment. The dependency is updated to evalport-sdk, the backward-compatible openeval.* imports are in place, and the lockfile has been regenerated. It looks ready for maintainer review.

[adhabnr-ux]

Update, and heads up: I just noticed AutoGen's README now says the project is in maintenance mode — "will not receive new features or enhancements," and contributions are limited to bug fixes, security patches, and documentation improvements going forward. That's almost certainly why this has sat without maintainer engagement; it's not a priority issue, it's a scope one.

Given that, I don't think it makes sense to keep waiting on a maintainer review here. Dresden — since the code itself is solid and tested, would you be open to us publishing it as a standalone autogen-openeval-adapter package instead, as I floated above? That way it's still usable by anyone on AutoGen without needing this repo to accept a new feature it's no longer positioned to take.

Separately, I'll look at whether Microsoft Agent Framework (the active successor) is a better long-term home for this kind of integration. Thanks for the solid work on this either way — appreciate you sticking with it.

[adhabnr-ux]

Closing the loop on this: the standalone package is published now — autogen-openeval-adapter, built directly from this PR's to_openeval()/from_openeval() design (round-trip tested, validated against the real EvalPort spec, 6/6 tests passing).

Source: https://github.com/adhabnr-ux/evalport/tree/main/adapters/autogen-openeval-adapter
Install (once published to PyPI): pip install autogen-openeval-adapter

I also opened a proposal for native support in Microsoft Agent Framework, since that's the actively maintained successor: microsoft/agent-framework#7590

Dresden — full credit to your work here, the README explicitly credits this PR as the design origin. This PR can stay open as the historical record / reference, but I don't think there's anything actionable left for AutoGen itself given the maintenance-mode constraints. Thanks again for the clean implementation.

[adhabnr-ux]

Hi Eric Zhu (@ekzhu) — following up here since this PR has been open since Jul 30 with no maintainer review yet, and it's currently unlabeled/unassigned. Quick context: this adds to_evalport()/from_evalport() so AutoGen eval tasks round-trip through EvalPort, an open interchange format for LLM eval data (already integrated into UK AISI's Inspect AI, published on PyPI + npm). Dresden (@DresdenGman) has kept the branch current and flagged it ready for review twice now (most recently Aug 8). Would you, or whoever's the right person for autogen-ext reviews, be able to take a look — even just to confirm direction before it goes stale? Happy to answer anything on the EvalPort side.

[adhabnr-ux]

Correcting myself — my last comment tagging Eric Zhu (@ekzhu) was out of date with my own earlier comment on this same thread. To be clear for anyone reading this later: given AutoGen's maintenance-mode status, we're not actually expecting/needing this PR to merge — the standalone autogen-openeval-adapter package (linked above) already covers this, and I've opened a fresh proposal for native support in the actively-developed successor instead (microsoft/agent-framework#7590). No action needed here from a maintainer; this PR can stay open purely as the historical reference for the design. Apologies for the noise.