Dataset Viewer, Structure and Libraries docs #1070

lhoestq · 2023-10-31T15:58:44Z

The goal is to make the Dataset Hub docs more focused on the Hub features like the Viewer, and less on the datasets lib. In this PR datasets becomes one of the many libraries that can be use with the Hub instead of being shown as en entry point.

I added a Configure the Dataset Viewer page

-> this will help users have a working Dataset Viewer without knowledge about how datasets works
I added a Data files Configuration page

-> this gives more detail on how to structure a dataset (e.g. for splits)
I added a Libraries page and dedicated pages for:
- Dask
- DuckDB
- Datasets (redirects to the datasets docs)
- Pandas
- WebDataset
-> the focus is less on the datasets library to show that people are actually free to use whatever tools they want.
-> they're pretty simple for now and we should keep enriching them
Added Uploading datasets and Downloading datasets for consistency with model docs

TODO in the datasets docs (will open a PR shortly):

remove "create a dataset" docs content and redirect to the one on the Hub instead
remove "repository structure" docs content and redirect to the one on the Hub instead

HuggingFaceDocBuilderDev · 2023-10-31T16:03:26Z

The documentation is not available anymore as the PR was closed or merged.

lhoestq · 2023-11-02T17:13:08Z

The docs preview doesn't work (internal report here) but feel free to start reviewing on GitHub :)

also cc @julien-c if you want to check the docs I'm adding to the Datasets section of the Hub documentation

severo

Thanks a lot for these docs, they are at the correct level of simplicity and avoid having to understand the details of the datasets library.

docs/hub/datasets-adding.md

docs/hub/datasets-dask.md

docs/hub/datasets-usage.md

docs/hub/datasets-libraries.md

docs/hub/datasets-duckdb.md

docs/hub/datasets-webdataset.md

julien-c

can you ping me again when the doc-build worked? 😅

docs/hub/datasets-data-files-configuration.md

julien-c · 2023-11-02T19:55:31Z

docs/hub/datasets-libraries.md

from a quick glance i do like the fact that we present multiple tools that are compatible with dataset repos! That's quite cool

julien-c

one last comment, you should also tag @davanstrien for review IMO given the recent post https://huggingface.co/blog/researcher-dataset-sharing and interests in dataset advocacy!!

Wauplin · 2023-11-03T09:02:25Z

docs/hub/datasets-adding.md

+
+### Create a Dataset card
+
+Adding a Dataset card is super valuable for helping users find your dataset and understand how to use it responsibly.


I would add a link to https://huggingface.co/docs/hub/datasets-cards for users that don't know what a Dataset card is.

docs/hub/datasets-adding.md

docs/hub/datasets-dask.md

Co-authored-by: Sylvain Lesage <sylvain.lesage@huggingface.co> Co-authored-by: Lucain <lucainp@gmail.com> Co-authored-by: Julien Chaumond <julien@huggingface.co>

davanstrien · 2023-11-06T12:08:17Z

Question, do we also want to add Argilla + Prodigy to the overview table in docs/hub/datasets-libraries.md. Could otherwise be worth adding a section on annotation tools that intergrate with the Hub?

I understand it will be in another PR. See #1070 (comment)

Sorry, missed that :)

mariosasko

And some comments from me:

docs/hub/_toctree.yml

mariosasko · 2023-11-06T17:22:39Z

docs/hub/datasets-adding.md

+Some libraries like [🤗 Datasets](https://huggingface.co/docs/datasets/index), [Pandas](https://pandas.pydata.org/), [Dask](https://www.dask.org/) or [DuckDB](https://duckdb.org/) can upload files to the Hub.
+See the list of [Libraries supported by the Datasets Hub](./datasets-libraries) for more information.


Would be better to have links to the upload examples from these libraries' dedicated doc pages here (we can split these pages into the Upload and Download sections to make them linkable)

The links already appear on the navigation tab on the left when you are on this page

Okay, I see. But I'm not sure why these links are not expanded (automatically) when clicking on the [Libraries supported by the Datasets Hub](./datasets-libraries) link on my machine.

Hmm I'm also seeing this behavior when clicking on the Libraries link from the documentation index page.
Let's see with the docs front-end team if we can fix that

docs/hub/datasets-viewer.md

docs/hub/datasets-viewer-configure.md

docs/hub/_toctree.yml

Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com>

lhoestq · 2023-11-06T21:39:05Z

Thanks for the comments, I took them into account :)

polinaeterna

very much like! i also added a few suggestions :)

docs/hub/datasets-adding.md

docs/hub/datasets-dask.md

docs/hub/datasets-adding.md

docs/hub/datasets-viewer.md

docs/hub/datasets-data-files-configuration.md

polinaeterna · 2023-11-07T14:46:28Z

docs/hub/datasets-file-names-and-splits.md

+
+## Basic use-case
+
+If your dataset isn't split into [train/validation/test splits](https://en.wikipedia.org/wiki/Training,_validation,_and_test_data_sets), the simplest dataset structure is to have one file: `data.csv` (this works with any supported file format and any file name).


any supported data format - there are a lot of mentions of supported formats through all the docs so maybe indeed it makes sense to list a complete set of them somewhere and point to it?

docs/hub/datasets-file-names-and-splits.md

docs/hub/datasets-manual-configuration.md

Co-authored-by: Polina Kazakova <polina@huggingface.co>

lhoestq · 2023-11-09T16:16:46Z

Let me know if you have other comments @mariosasko @polinaeterna @Wauplin

Also cc @julien-c the documentation preview is working now if you want to take a look

Wauplin

Very nice!
Made a last check on the docs and everything looks fine 👍

docs/hub/datasets-file-names-and-splits.md

Co-authored-by: Lucain <lucainp@gmail.com>

mariosasko

Thanks, LGTM!

polinaeterna

thank you a lot for this work! i've left a couple of nits suggestions, you can ignore them except for the link to the file-formats section (otherwise it's broken) and some punctuation.

docs/hub/datasets-file-names-and-splits.md

docs/hub/datasets-manual-configuration.md

docs/hub/datasets-pandas.md

docs/hub/datasets-viewer-configure.md

docs/hub/datasets-dask.md

docs/hub/datasets-duckdb.md

julien-c · 2023-11-14T18:49:38Z

let's 🚢 this verrrrry long-standing PR no?

Co-authored-by: Polina Kazakova <polina@huggingface.co>

lhoestq · 2023-11-15T11:51:21Z

Thanks for all the reviews :)

more datasets docs

e60a969

lhoestq added 5 commits October 31, 2023 17:06

add configure your dataset

4d7bc4a

minor

0f7a201

minor

de191ca

update toc

1216878

minor

441c8cc

lhoestq marked this pull request as ready for review October 31, 2023 17:04

lhoestq marked this pull request as draft October 31, 2023 17:08

add dataset structure docs

c9e24a0

lhoestq changed the title ~~Dataset Viewer and Libraries docs~~ Dataset Viewer, Structure and Libraries docs Oct 31, 2023

rename sections

c4e5ed0

lhoestq marked this pull request as ready for review November 2, 2023 17:12

lhoestq requested review from Wauplin, mariosasko and severo November 2, 2023 17:13

severo reviewed Nov 2, 2023

View reviewed changes

julien-c reviewed Nov 2, 2023

View reviewed changes

docs/hub/datasets-data-files-configuration.md Outdated Show resolved Hide resolved

julien-c reviewed Nov 2, 2023

View reviewed changes

Wauplin reviewed Nov 3, 2023

View reviewed changes

lhoestq and others added 7 commits November 3, 2023 16:48

Apply suggestions from code review

3a50846

Co-authored-by: Sylvain Lesage <sylvain.lesage@huggingface.co> Co-authored-by: Lucain <lucainp@gmail.com> Co-authored-by: Julien Chaumond <julien@huggingface.co>

sylvain's comments: Adding a new dataset

0d1e1f4

sylvain-s comments: Configure the Dataset Viewer

b1b6f5b

sylvain's comments: File names and splits

1e2923a

sylvain's comments: Manual Configuration

07c493e

sylvain's comments: Libraries

10fc614

sylvain's comment: login

624c0c4

mariosasko reviewed Nov 6, 2023

View reviewed changes

lhoestq and others added 3 commits November 6, 2023 22:00

Apply suggestions from code review

7bf18c5

Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com>

rename titles for consistency

3e4836b

Add Downloading Datasets

1b4b81a

polinaeterna reviewed Nov 7, 2023

View reviewed changes

lhoestq and others added 10 commits November 8, 2023 13:58

Apply suggestions from code review

e6d7d39

Co-authored-by: Polina Kazakova <polina@huggingface.co>

Merge branch 'more-datasets-docs-continued' into more-datasets-docs

4b74ddc

more links to list of supported formats

0028030

move supported file formats to upload page

fe1e75e

fix link to row

e257db7

fix links

79d2812

again

2ffc2cc

remove duplicate

c9d0f53

move configure viewer to bottom of page

dac5618

fix to_parquet example

74dbd4f

severo mentioned this pull request Nov 9, 2023

Improve error message when no suitable file is found huggingface/dataset-viewer#2082

Closed

minor fix

8e3da3c

Wauplin approved these changes Nov 14, 2023

View reviewed changes

docs/hub/datasets-file-names-and-splits.md Outdated Show resolved Hide resolved

Update docs/hub/datasets-file-names-and-splits.md

e63d0e4

Co-authored-by: Lucain <lucainp@gmail.com>

mariosasko approved these changes Nov 14, 2023

View reviewed changes

polinaeterna approved these changes Nov 14, 2023

View reviewed changes

Apply suggestions from code review

fa1093e

Co-authored-by: Polina Kazakova <polina@huggingface.co>

lhoestq merged commit 35884ae into main Nov 15, 2023

lhoestq deleted the more-datasets-docs branch November 15, 2023 11:51

lhoestq mentioned this pull request Nov 15, 2023

Fix broken links #1099

Merged


		### Create a Dataset card

		Adding a Dataset card is super valuable for helping users find your dataset and understand how to use it responsibly.

		Some libraries like [🤗 Datasets](https://huggingface.co/docs/datasets/index), [Pandas](https://pandas.pydata.org/), [Dask](https://www.dask.org/) or [DuckDB](https://duckdb.org/) can upload files to the Hub.
		See the list of [Libraries supported by the Datasets Hub](./datasets-libraries) for more information.


		## Basic use-case

		If your dataset isn't split into [train/validation/test splits](https://en.wikipedia.org/wiki/Training,_validation,_and_test_data_sets), the simplest dataset structure is to have one file: `data.csv` (this works with any supported file format and any file name).

Dataset Viewer, Structure and Libraries docs #1070

Dataset Viewer, Structure and Libraries docs #1070

Uh oh!

Conversation

lhoestq commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lhoestq commented Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

severo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julien-c left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

julien-c Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

julien-c left a comment

Choose a reason for hiding this comment

Uh oh!

Wauplin Nov 3, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

davanstrien commented Nov 6, 2023

Uh oh!

mariosasko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mariosasko Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

lhoestq Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

mariosasko Nov 8, 2023

Choose a reason for hiding this comment

Uh oh!

lhoestq Nov 9, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lhoestq commented Nov 6, 2023

Uh oh!

polinaeterna left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

polinaeterna Nov 7, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lhoestq commented Oct 31, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 31, 2023 •

edited

Loading

lhoestq commented Nov 2, 2023 •

edited

Loading

lhoestq commented Nov 9, 2023 •

edited

Loading