Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement to_dict and to_pandas for Dataset #1889

Merged
merged 7 commits into from
Feb 18, 2021
Merged

Implement to_dict and to_pandas for Dataset #1889

merged 7 commits into from
Feb 18, 2021

Conversation

SBrandeis
Copy link
Contributor

With options to return a generator or the full dataset

lhoestq
lhoestq reviewed Feb 17, 2021
View reviewed changes
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thanks !

Also can you use query_table with self._indices (same change as I suggested in the to_csv PR) ? This way this will support shuffled/sharded datasets

src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
lhoestq
lhoestq reviewed Feb 18, 2021
View reviewed changes
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice ! Thanks for adding the support for datasets with an indices mapping.
I added some more comments about query_table and the batched=False case, as well as suggestions to use len(self) instead of self._data.num_rows in query_table

src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
src/datasets/arrow_dataset.py Outdated Show resolved Hide resolved
tests/test_arrow_dataset.py Show resolved Hide resolved
lhoestq
lhoestq approved these changes Feb 18, 2021
View reviewed changes
Copy link
Member

@lhoestq lhoestq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice thank you !

@lhoestq lhoestq merged commit 9acb9da into huggingface:master Feb 18, 2021
@lhoestq
Copy link
Member

lhoestq commented Feb 18, 2021

Next step is going to add these two in the documentation ^^

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lhoestq lhoestq lhoestq approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@SBrandeis @lhoestq