Enable the private datasets

[severo]

The code is already present to pass the token, but it's disabled in the code (hardcoded):

https://github.com/huggingface/datasets-preview-backend/blob/df04ffba9ca1a432ed65e220cf7722e518e0d4f8/src/datasets_preview_backend/cache.py#L119-L120

• enable private datasets and manage their cache adequately
• separate private caches from public caches: for authenticated requests, we need to check every time, or at least use a much lower TTL, because an access can be removed. Also: since a hub dataset can be turned private, how should we manage them?
• add doc. See f6576d5

[severo]

Previous code has been removed with b08e649, for reference

[severo]

Requested here: huggingface/datasets#3604

[severo]

Also: asked for by the AutoNLP team, to be able to preview the training datasets

[severo]

Also requested here: https://huggingface.slack.com/archives/C01229B19EX/p1647460106784919?thread_ts=1647372809.022069&cid=C01229B19EX

[severo]

https://github.com/huggingface/moon-landing/pull/2442 has added an endpoint to check the authentication:

https://github.com/huggingface/moon-landing/blob/bb1bcde43561c1831aef4c1c6a331dec7f735171/server/server.ts#L368-L369

[rajshah4]

As we get more enterprise prospects and customers, it would be nice to be able to show off the dataset viewer on private datasets. So +1 on this feature request from me.

[severo]

Yes, it's in the list of the next features that we will implement. We're working on a roadmap to make it clearer for everybody what could be expected in the following months. Viewer for private models could not be reasonably implemented with the previous "artisanal" infrastructure because it would have fallen under the load, but now that we run in Kubernetes, it should work seamlessly.

[severo]

Required here for private hub: https://huggingface.slack.com/archives/CTKK32GE8/p1658236535144079?thread_ts=1658236048.820219&cid=CTKK32GE8

[severo]

Priority level:

https://huggingface.slack.com/archives/C03M773CSBS/p1663233611753009?thread_ts=1662756193.298359&cid=C03M773CSBS

it's not a top priority from what we've said yesterday (focus on shipping the dataset-server)

[severo]

Requested on the forum: https://discuss.huggingface.co/t/the-dataset-preview-has-been-disabled-on-this-dataset/21339/5

[severo]

Also here: https://discuss.huggingface.co/t/does-the-rest-api-work-with-private-repo/28987 (to consume the API)

[faustusdotbe]

Hi! Chiming in to show interest. I completely understand this is not a top priority, but being able to have the viewer in private datasets would be super cool (also as a last final "sanity check" before making the repo public for example).

Happy to help debug, if needed

[alexgshaw]

+1 to this feature

[DavidFarago]

+1 for this feature

[arikanev]

+1

[TheAnimeGuru]

+1 This would be very useful

[nealchandra]

I see this issue has been tagged as a P2 -- are you able to give us a rough estimate of what that might mean in terms of when it could land?

[severo]

No, we will update here when we have an ETA. Meanwhile, see #39 (comment)

[Stillerman]

+1

[prassanna-ravishankar]

Is there a way we can explicitly enable this for "private datasets"(at our own risk) ? At the moment my dataset has to be private, but I am concerned less about leakages etc. Would love to enable it

[severo]

Is there a way we can explicitly enable this for "private datasets"(at our own risk) ?

No, it's not possible at the moment. It is not a replacement, but if it works for you, maybe you can set your dataset as gated: the dataset viewer works with them.

[DavidFarago]

What is the strictest possible gating? Can you restrict it to effectively be a private dataset?

[severo]

What is the strictest possible gating? Can you restrict it to effectively be a private dataset?

It's not the same as private, but if you opt to manually approve the requests to access the gated dataset (https://huggingface.co/docs/hub/datasets-gated#manual-approval), you would avoid giving public access to the data.

[enoriega]

+1 to enable dataset viewer on private datasets

[eduardm]

+1. As an enterprise user, I don't want to share the dataset publicly, but not having a preview is really frustrating and makes collaboration difficult. Currently, we are creating a tiny dataset (10 rows) and making that public just to have a preview and understand what is what.

[andreemic]

+1 for this!

The datasets filtering and sorting UI is very very barebones, so the viewer would help to navigate private datasets a lot

[severo]

Also, internal request: https://huggingface.slack.com/archives/C02EMARJ65P/p1702930353945389

[qionghuang6]

+1 for viewing private datasets

[julien-c]

please continue +1'ing this issue (can be on the OP) so we get a sense for how many people/teams need this! 🙏

[sutgeorge]

+1

[egrace479]

This would be great for development of datasets, especially since the documentation for how the viewer/preview works is quite limited and currently requires piecing together information from multiple pages on the docs.

While we're waiting on this to be developed (assuming it still will be), could the docs be updated with clearer requirements for the preview feature (i.e., more than just that the dataset needs to be public)? When I run the list parquet files query I'm able to see that the parquet has failed, but it's not clear why. Is there a way to get more information about what went wrong? For one of the datasets in question it is simply a collection of images in train and val folders that is simpler than a similar dataset for which the parquet conversion was successful and displays a preview.

[severo]

Done!

The private datasets are now supported in the datasets-server, enabling the dataset viewer + parquet conversion on the dataset pages.

Enregistrement.de.l.ecran.2024-01-31.a.10.39.59.mov

Note that it's a paid feature, available as of today for Pro users and Enterprise orgs. Please give us feedback when you try this new feature!

[faustusdotbe]

Merci Sylvain !