Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Enable isolated node eval for answer generator nodes (incl. OpenAI Node) #3036

Merged
merged 4 commits into from Aug 14, 2022

Conversation

tholor
Copy link
Member

@tholor tholor commented Aug 12, 2022

Related Issues

Proposed Changes:

Making the BaseGenerator 's run method more similar to the one from BaseReader by

  • adding a param add_isolated_node_eval
  • in case the mode is active, taking the same approach as in BaseReader here. The only difference here is that we are not adding the meta data of documents, as the answers are not extracted from any direct docs but are generated. So a direct link here seems impropropriate from my perspective

How did you test it?

  • manual test by replacing the reader in our Tutorial 5 by: reader = OpenAIAnswerGenerator(api_key="<api-key>")

Notes for the reviewer

  • Let me know if you also agree, that the metadata linking (as done in the BaseReader) doesn't make sense here for the generator
  • Just touched the run() method here. Didn't check if run_batch also needs a fix or how the eval mode in batch mode currently works.
  • Don't see big side effects possible here as the new if block is quite isolated from anything else

Checklist

@tholor tholor requested a review from a team as a code owner August 12, 2022 16:05
Copy link
Member

@tstadel tstadel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Just had to fix a small mypy complaint.

@tstadel tstadel merged commit 1b422ab into master Aug 14, 2022
@tstadel tstadel deleted the enable_isolated_eval_generators branch August 14, 2022 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable isolated_node_eval for AnswerGenerator nodes (incl. OpenAI's)
3 participants