[Dataset] Remove support for DatasetDict as input into from_huggingface()
#37555
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
scottjlee
requested review from
richardliaw,
gjoliver,
krfricke,
xwjiang2010,
amogkam,
matthewdeng,
Yard1,
maxpumperla,
a team,
ericl,
scv119,
c21,
bveeramani and
raulchen
as code owners
amogkam
reviewed
Jul 19, 2023
python/ray/data/read_api.py
Outdated
| dataset: A `Hugging Face Datasets Dataset`_ or `DatasetDict`_. | ||
| :class:`~ray.data.IterableDataset` isn't supported. | ||
| dataset: A `Hugging Face Datasets Dataset`_. | ||
| :class:`~ray.data.IterableDataset` and |
There was a problem hiding this comment.
Man this IterableDataset link is just straight up wrong 😅. Not sure how this had slipped in before.
| if isinstance(dataset, datasets.DatasetDict): | ||
| available_keys = list(dataset.keys()) | ||
| logger.warning( | ||
| "You provided a Huggingface DatasetDict which contains multiple " |
There was a problem hiding this comment.
Let's keep this, but change to an error message
There was a problem hiding this comment.
would the last case raising TypeError cover this case as well? or do we want to explicitly raise a separate error for this case, with a message like "DatasetDict is not supported"?
There was a problem hiding this comment.
I think we want an explicit error for this case since we can show the actual splits and provide an example on how to get a specific split from the DatasetDict.
There was a problem hiding this comment.
Btw, it should be a DeprecationWarning raised since technically this is a breaking change right.
Signed-off-by: Scott Lee <sjl@anyscale.com>
c21
approved these changes
Jul 19, 2023
python/ray/data/read_api.py
Outdated
| "Datasets. To convert just a single Huggingface Dataset to a " | ||
| logger.error( | ||
| "You provided a Hugging Face DatasetDict which contains multiple " | ||
| "datasets, but `from_huggingface` now only accepts a siingle Hugging Face " |
amogkam
requested changes
Jul 19, 2023
python/ray/data/read_api.py
Outdated
| "You provided a Huggingface DatasetDict which contains multiple " | ||
| "datasets. The output of `from_huggingface` is a dictionary of Ray " | ||
| "Datasets. To convert just a single Huggingface Dataset to a " | ||
| logger.error( |
There was a problem hiding this comment.
don't we want to raise an exception here?
There was a problem hiding this comment.
ah okay, got it. let me raise a TypeError here
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
amogkam
approved these changes
Jul 20, 2023
NripeshN
pushed a commit
to NripeshN/ray
that referenced
this pull request
Aug 15, 2023
…gface()` (ray-project#37555) To keep the from_huggingface() API simple and consistent with other from_xxx read APIs, we remove support for Hugging Face datasets.DatasetDict, so that the method only accepts a Hugging Face datasets.Dataset. --------- Signed-off-by: Scott Lee <sjl@anyscale.com> Signed-off-by: NripeshN <nn2012@hw.ac.uk>
harborn
pushed a commit
to harborn/ray
that referenced
this pull request
Aug 17, 2023
…gface()` (ray-project#37555) To keep the from_huggingface() API simple and consistent with other from_xxx read APIs, we remove support for Hugging Face datasets.DatasetDict, so that the method only accepts a Hugging Face datasets.Dataset. --------- Signed-off-by: Scott Lee <sjl@anyscale.com> Signed-off-by: harborn <gangsheng.wu@intel.com>
harborn
pushed a commit
to harborn/ray
that referenced
this pull request
Aug 17, 2023
…gface()` (ray-project#37555) To keep the from_huggingface() API simple and consistent with other from_xxx read APIs, we remove support for Hugging Face datasets.DatasetDict, so that the method only accepts a Hugging Face datasets.Dataset. --------- Signed-off-by: Scott Lee <sjl@anyscale.com>
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…gface()` (ray-project#37555) To keep the from_huggingface() API simple and consistent with other from_xxx read APIs, we remove support for Hugging Face datasets.DatasetDict, so that the method only accepts a Hugging Face datasets.Dataset. --------- Signed-off-by: Scott Lee <sjl@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
…gface()` (ray-project#37555) To keep the from_huggingface() API simple and consistent with other from_xxx read APIs, we remove support for Hugging Face datasets.DatasetDict, so that the method only accepts a Hugging Face datasets.Dataset. --------- Signed-off-by: Scott Lee <sjl@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
To keep the
from_huggingface()API simple and consistent with otherfrom_xxxread APIs, we remove support for Hugging Facedatasets.DatasetDict, so that the method only accepts a Hugging Facedatasets.Dataset.Related issue number
Closes #37523
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.