-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add EvolComplexity
and EvolQuality
#299
feat: Add EvolComplexity
and EvolQuality
#299
Conversation
docs are still WIP @plaguss |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Just a reminder in the docstrings for one of the references. The only thing I'm not sure is if it makes sense to keep both EvolInstruct
and EvolComplexity
, to avoid complexity for the users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi! Looks good to me, I left some comments before merging
Also, could we add a unit test covering the Evol Quality Task use of the method |
chore: re-use strings in tests chore: outlined initial tests chore: rename Evol*GeneratorTask to Evol*Task
@@ -120,6 +116,16 @@ def test_evol_task(evolution_method: str, expected: str, instruct_type: object): | |||
assert isinstance(task, instruct_type) | |||
assert task.system_prompt == "" | |||
if isinstance(expected, str): | |||
print("---COSA") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is happening here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ups, forgot to remove those, it's hard to find the differences from the prompts directly from pytest
Here it is: 7c309aa. If the tests pass, and the docs are correct, I'll merge it |
Added
EvolComplexityTask
andEvolQualityTask
.EvolComplexityTask
is focused on improving the complexity of instructions and is a special implementation ofEvolInstruct
where we don't allow for thebreadth
evolution method.EvolQualityTask
is focused on improving the complexity ofgeneration
/responses and is a more altered implementation ofEvolInstruct
with evolution methods for["helpfulness", "relevance", "depth", "creativity", "details"]
.Closes #296
Closes #299