Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add multiprocess dependency #467

Merged
merged 1 commit into from
Mar 23, 2024
Merged

Add multiprocess dependency #467

merged 1 commit into from
Mar 23, 2024

Conversation

gabrielmbmb
Copy link
Member

Description

This PR adds multiprocess as a dependency of distilabel to replace the builtin library multiprocessing, as the serialization is better thanks to dill, and works better on Jupyter Notebooks.

@gabrielmbmb gabrielmbmb added the enhancement New feature or request label Mar 23, 2024
@gabrielmbmb gabrielmbmb added this to the 1.0.0 milestone Mar 23, 2024
@gabrielmbmb gabrielmbmb self-assigned this Mar 23, 2024
@plaguss plaguss merged commit 8682a78 into core-refactor Mar 23, 2024
4 checks passed
@plaguss plaguss deleted the use_multiprocess branch March 23, 2024 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@plaguss plaguss plaguss approved these changes

@alvarobartt alvarobartt Awaiting requested review from alvarobartt

Assignees

@gabrielmbmb gabrielmbmb

Labels
enhancement New feature or request
Projects
⚗️ distilabel
Status: Done
Milestone
1.0.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@gabrielmbmb @plaguss