Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFC] Run Evaluator Chains #5618

Closed
wants to merge 3 commits into from
Closed

[RFC] Run Evaluator Chains #5618

wants to merge 3 commits into from

Conversation

vowelparrot
Copy link
Contributor

@vowelparrot vowelparrot commented Jun 2, 2023 •
edited

Will try to land #5750 first
Pros:

  • Fits many but not all feedback applications.
  • Maintains a flexible base with configurable implementations.
  • Doesn't implement THAT many new classes for the core logic

Cons:

  • doesn't handle pairwise/group evals
  • trajectory evals are a bit hacky ( you have to remember to turn on when tracing rather than building it from the run object)
  • possibly too opinionated by forcing an input mapper and output parser?
  • Doesn't (yet) allow easy composability of the reasoning step (most do COT built in)
  • Most of the scores are binary - no nuance or more advanced logic. Doesn't (yet) decouple the scoring logic from the core of the prompt
  • Not sure if the creation functions are actually much better than creating a child and making a complicated constructor

Example making 5 different evaluators:

image

@vowelparrot vowelparrot mentioned this pull request Jun 2, 2023
Merged
vowelparrot added a commit to langchain-ai/langsmith-sdk that referenced this pull request Jun 2, 2023
@vowelparrot @nfcampos
Add RunEvaluator to Py and JS Libraries (#4)
0159529 
See: langchain-ai/langchain#5618 for example
LangChain implementations


I think we'll want some simple non-langchain completion function
evaluators if we want this core interface to be of much use outside the
OSS project. I also don't think I want to land the StringEvaluator class
but put it up as an example of one approach (that contrasts with the
ones above)

Co-authored-by: Nuno Campos <nuno@boringbits.io>
@vowelparrot vowelparrot mentioned this pull request Jun 5, 2023
Merged
@vowelparrot vowelparrot closed this Jun 5, 2023
vowelparrot added a commit that referenced this pull request Jun 6, 2023
@vowelparrot
Base RunEvaluator Chain (#5750)
217b5cc 
Clean up a bit and only implement the QA and reference free
implementations from #5618
Undertone0809 pushed a commit to Undertone0809/langchain that referenced this pull request Jun 19, 2023
@vowelparrot @Undertone0809
Base RunEvaluator Chain (langchain-ai#5750)
10332e3 
Clean up a bit and only implement the QA and reference free
implementations from langchain-ai#5618
kacperlukawski pushed a commit to kacperlukawski/langchain that referenced this pull request Jun 29, 2023
@vowelparrot @kacperlukawski
Base RunEvaluator Chain (langchain-ai#5750)
e68dcf9 
Clean up a bit and only implement the QA and reference free
implementations from langchain-ai#5618
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@vowelparrot