fix(py): type evaluator callbacks as BaseEvalDataPoint - #5995
Conversation
Mirror JS: EvaluatorFn receives BaseEvalDataPoint with a guaranteed test_case_id after the stepper fill-in, so custom evaluators need not defensively coalesce an optional id.
There was a problem hiding this comment.
Code Review
This pull request refactors the evaluator typing system to ensure that evaluator callbacks receive a BaseEvalDataPoint with a guaranteed test_case_id instead of a BaseDataPoint where the ID is optional. A helper function _as_eval_datapoint has been introduced to automatically generate a UUID for missing test case IDs without mutating the original dataset. Corresponding updates have been made across various plugins, tests, and samples. There are no review comments, so I have no feedback to provide.
|
@jeffdh5 ready for review when you have a moment. This mirrors the JS |
Summary
EvaluatorFnto takeBaseEvalDataPoint(test_case_id: str) instead ofBaseDataPoint(test_case_idoptional), matching the JS evaluator API.BaseEvalDataPointin the stepper (UUID fill-in) before calling the callback, without mutating the datasetBaseDataPoint.genkit-evaluators, google-genai Vertex evaluators) and the evaluators sample to use the required id directly.Fixes #5991
cc @jeffdh5
Test plan
pytestevaluator helper + veneer eval +genkit-evaluatorstests (10 passed)datapoint.test_case_idwithoutor ''under the typechecker