Skip to content

fix(py): type evaluator callbacks as BaseEvalDataPoint - #5995

Open
a2105z wants to merge 1 commit into
genkit-ai:mainfrom
a2105z:fix/py-evaluator-base-eval-datapoint
Open

fix(py): type evaluator callbacks as BaseEvalDataPoint#5995
a2105z wants to merge 1 commit into
genkit-ai:mainfrom
a2105z:fix/py-evaluator-base-eval-datapoint

Conversation

@a2105z

@a2105z a2105z commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Change Python EvaluatorFn to take BaseEvalDataPoint (test_case_id: str) instead of BaseDataPoint (test_case_id optional), matching the JS evaluator API.
  • Construct a BaseEvalDataPoint in the stepper (UUID fill-in) before calling the callback, without mutating the dataset BaseDataPoint.
  • Update first-party evaluators (genkit-evaluators, google-genai Vertex evaluators) and the evaluators sample to use the required id directly.
  • Add regression tests for id fill-in / non-mutation and callback typing.

Fixes #5991

cc @jeffdh5

Test plan

  • pytest evaluator helper + veneer eval + genkit-evaluators tests (10 passed)
  • Confirm a custom evaluator can use datapoint.test_case_id without or '' under the typechecker

Mirror JS: EvaluatorFn receives BaseEvalDataPoint with a guaranteed
test_case_id after the stepper fill-in, so custom evaluators need not
defensively coalesce an optional id.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the evaluator typing system to ensure that evaluator callbacks receive a BaseEvalDataPoint with a guaranteed test_case_id instead of a BaseDataPoint where the ID is optional. A helper function _as_eval_datapoint has been introduced to automatically generate a UUID for missing test case IDs without mutating the original dataset. Corresponding updates have been made across various plugins, tests, and samples. There are no review comments, so I have no feedback to provide.

@a2105z

a2105z commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

@jeffdh5 ready for review when you have a moment.

This mirrors the JS EvaluatorFn / BaseEvalDataPoint split, and I reconfirmed the evaluator tests pass locally (helper + veneer eval + genkit-evaluators). Happy to adjust anything.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

py: type evaluator callbacks as BaseEvalDataPoint

1 participant