Skip to content

Conversation

wengh
Copy link
Collaborator

@wengh wengh commented Jan 31, 2025

Overview

This PR improves usability by grouping the HuggingFace source and sink data sources under the same name huggingface by adding a wrapper class.

IMPORTANT: Skip the first commit (renaming huggingface.py) when you review. Otherwise the renaming messes up GitHub's PR diff.

Link to diff that excludes the first commit

What's changed

  • Rename huggingface.py to huggingface_source.py and change the class name to HuggingFaceSource.
  • Add a new HuggingFaceDatasets data source class that delegates to either HuggingFaceSource or HuggingFaceSink based on the requested operation.
  • Update tests to use the new name.

@wengh wengh changed the base branch from main to haoyu-custom-split January 31, 2025 00:35
@wengh wengh marked this pull request as ready for review January 31, 2025 00:45
@wengh wengh marked this pull request as draft January 31, 2025 00:46
@wengh wengh changed the base branch from haoyu-custom-split to main January 31, 2025 00:48
@wengh wengh marked this pull request as ready for review January 31, 2025 00:48
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great !

@wengh wengh merged commit 8647b7d into main Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants