Idea: EvalPort JSON (LLM eval result sets) as a GX-validatable batch

[adhabnr-ux]

Hi GX team — this is a lightweight integration idea, not a feature request, so posting under Integrations rather than opening an RFC.

EvalPort (https://github.com/adhabnr-ux/evalport) is an open spec + JSON Schema for portable LLM-eval artifacts: test suites, test cases, and result sets — basically "a batch of eval-run rows, one per test case, with a score/pass field." That's data GX already knows how to eat — a flat-ish list of typed rows — it just happens to come from an LLM eval run instead of a database table.

I looked at great_expectations/core/expectation_suite.py (ExpectationSuite — name, expectations, suite_parameters, meta) and the checkpoint/validator flow, and the fit isn't a new Expectation type — it's just documenting a recipe: load an EvalPort ResultSet JSON (an array of {test_case_id, score, passed, ...} rows, validated against the schemas at https://github.com/adhabnr-ux/evalport/tree/main/spec/schemas) as a pandas/JSON batch, then gate CI with existing GX primitives, e.g.:

suite = context.add_expectation_suite("evalport_regression_gate")
suite.add_expectation_configuration(...)  # expect_column_values_to_be_between("score", min_value=0.8)
validation_result = context.get_validator(batch_request=..., expectation_suite=suite).validate()

i.e. regression-gate an eval run's per-case or aggregate scores the same way you'd gate a data-quality check, with zero changes to GX core.

Happy to write this up as a short example/notebook if there's any interest — otherwise feel free to ignore, just wanted to float it since the fit looked natural from the code. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md