Skip to content

[Evals] Consistent testCaseId, support patch updates #975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Oct 11, 2024
Merged

[Evals] Consistent testCaseId, support patch updates #975

merged 12 commits into from
Oct 11, 2024

Conversation

@ssbushi
Copy link
Contributor

@ssbushi ssbushi commented Sep 26, 2024 •
edited
Loading

In this PR:

  • Consistent and persistent testCaseId throughout evalsland. This facilitates partial dataset update.
  • UpdateDataset now supports partial updates to dataset.
  • Refactor EvalFlowInput -> EvalInferenceInput
  • Minor changes to runEvaluation API
  • make EvaluatorFn always get testCaseId, enforcing consistency with testCaseId in plugins.
  • JSDoc updates + tests

Checklist:

  • Tested (manually, unit tested)

@ssbushi ssbushi marked this pull request as ready for review September 30, 2024 20:36
@ssbushi ssbushi merged commit 345f814 into next Oct 11, 2024
4 checks passed
@ssbushi ssbushi deleted the sb/targetAction branch October 11, 2024 19:13
@tonybaroneee
Copy link
Collaborator

tonybaroneee commented Oct 16, 2024 •
edited
Loading

@ssbushi FYI, whenever making changes to tools-common types that the Developer UI relies on, you need to follow these steps to update package/genkit-ai-tools-common.tgz. Without this, the UI might fail to compile (as it does now on next).

@ssbushi
Copy link
Contributor Author

ssbushi commented Oct 16, 2024

Sorry about that. I had https://github.com/FirebasePrivate/genkit-ui/pull/1075 in the works but did not push.

@tonybaroneee
Copy link
Collaborator

tonybaroneee commented Oct 16, 2024

No worries, looks good. Thanks!

@shrutip90 shrutip90 mentioned this pull request Oct 24, 2024
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pavelgj pavelgj pavelgj approved these changes

@shrutip90 shrutip90 Awaiting requested review from shrutip90

Assignees

No one assigned

Labels

None yet

Projects

Genkit Backlog
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ssbushi @tonybaroneee @pavelgj