-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add Evol instruct task #237
Conversation
if "sorry" in output.lower() and len(output.split(" ")) < 80: | ||
return | ||
|
||
# 3) The output only contains punctuation and stop words |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the stopwords comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think one of our current dependencies supports this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What dependency do you refer to?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we already by chance were downloading nltk
we might have been able to re-use their logic. Sadly, we don't
…evol-instruct-task
… method and redirect to the original paper
docs/technical-reference/tasks.md
Outdated
|
||
!!! note | ||
The original definition of `EvolInstruct` considers an elimination evolving step with different | ||
situations to remove the responses considered as failures. Section 3.2, *Elimination Evolving* in [WizardLM paper](https://arxiv.org/abs/2304.12244) shows these steps. We have implemented steps 2-4 as part of this task, but not step one. If the user wants to tackle this, an example can be seen in the following script: [examples/pipeline-openai-wizardl-equal-prompts.py](../../examples/pipeline-evol-instruct-alpaca.py). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the wrong file is referenced here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also mention more explicitly that it shows how to implement evaluation step 1 using distilabel please
Description
This PR adds a new text generation task: Evol-Instruct.
Closes #229