Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Evol instruct task #237

Merged
merged 15 commits into from
Jan 16, 2024
Merged

feat: Add Evol instruct task #237

merged 15 commits into from
Jan 16, 2024

Conversation

plaguss
Copy link
Contributor

@plaguss plaguss commented Jan 11, 2024

Description

This PR adds a new text generation task: Evol-Instruct.

Closes #229

if "sorry" in output.lower() and len(output.split(" ")) < 80:
return

# 3) The output only contains punctuation and stop words

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the stopwords comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think one of our current dependencies supports this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What dependency do you refer to?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we already by chance were downloading nltk we might have been able to re-use their logic. Sadly, we don't


!!! note
The original definition of `EvolInstruct` considers an elimination evolving step with different
situations to remove the responses considered as failures. Section 3.2, *Elimination Evolving* in [WizardLM paper](https://arxiv.org/abs/2304.12244) shows these steps. We have implemented steps 2-4 as part of this task, but not step one. If the user wants to tackle this, an example can be seen in the following script: [examples/pipeline-openai-wizardl-equal-prompts.py](../../examples/pipeline-evol-instruct-alpaca.py).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the wrong file is referenced here

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also mention more explicitly that it shows how to implement evaluation step 1 using distilabel please

@davidberenstein1957 davidberenstein1957 merged commit 28b60c1 into main Jan 16, 2024
4 checks passed
@davidberenstein1957 davidberenstein1957 deleted the evol-instruct-task branch January 16, 2024 15:56
@davidberenstein1957 davidberenstein1957 changed the title Add Evol instruct task feat: Add Evol instruct task Jan 16, 2024
@davidberenstein1957 davidberenstein1957 added this to the 0.4.0 milestone Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] add support for the EvolInstructTask from the WizardLM paper
2 participants