Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EvolComplexity task #415

Merged
merged 30 commits into from
Mar 18, 2024
Merged

Conversation

davidberenstein1957
Copy link
Member

@davidberenstein1957 davidberenstein1957 commented Mar 13, 2024

Description

Implements EvolComplexity as a wrapper around EvolInstruct but with limited template options.

Closes #392

Copy link
Member

@alvarobartt alvarobartt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't review yet, but I'd place evol_complexity under EvolInstruct since it's based on it, rather than having it in a separate module, we can work on the imports later on i.e. from distilabel.tasks import EvolComplexity, EvolInstruct

@davidberenstein1957
Copy link
Member Author

@alvarobartt, I thought it was more structured when searching for tasks to have them in separate folders but I'll change the structure a bit.

@davidberenstein1957 davidberenstein1957 marked this pull request as ready for review March 13, 2024 14:53
@davidberenstein1957
Copy link
Member Author

@alvarobartt feel free to review

Base automatically changed from evol-instruct-task to core-refactor March 16, 2024 11:10
Copy link
Member

@alvarobartt alvarobartt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi here! Some comments, but besides that I meant to nest the EvolComplexity within a separate directory as evol_instruct/complexity/base.py or something similar, to me is clearer, WDYT?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't mix the tests here, i.e. EvolInstruct separately

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll create a lot of code/test-duplication for inherently testing the same things so I think it is better to keep it like this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that as EvolComplexity is just inheriting from EvolInstruct and modifying the mutation_templates, only that should be tested. Because the rest of it is just a duplication from EvolInstruct, so there's no need to run the same tests twice, so a check on the mutation_templates should be enough IMO. That would also help on keeping everything more organized IMO.

src/distilabel/steps/task/evol_instruct/base.py Outdated Show resolved Hide resolved
src/distilabel/steps/task/evol_instruct/base.py Outdated Show resolved Hide resolved
src/distilabel/steps/task/evol_instruct/base.py Outdated Show resolved Hide resolved
@alvarobartt alvarobartt changed the title evol complexity task Add EvolComplexity task Mar 18, 2024
@alvarobartt alvarobartt linked an issue Mar 18, 2024 that may be closed by this pull request
@alvarobartt alvarobartt added this to the 1.0.0 milestone Mar 18, 2024
@davidberenstein1957 davidberenstein1957 merged commit f9d8380 into core-refactor Mar 18, 2024
4 checks passed
@davidberenstein1957 davidberenstein1957 deleted the feat/evol-complexity-task branch March 18, 2024 15:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

Adapt EvolComplexityTask to new Task interface
2 participants