[HfFileSystem] Less /paths-info calls

[lhoestq]

...or we get rate-limited in the dataset viewer (e.g. for FineWeb)

cc @severo

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

This is also causing bugs in datasets when loading datasets with many files, e.g. load_dataset('mteb/biblenlp-corpus-mmteb'):

huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/mteb/biblenlp-corpus-mmteb/paths-info/3912ed967b0834547f35b2da9470c4976b357c9a

could you take a look @Wauplin ? It would be cool to release this fix when you return :p

[Wauplin]

Conceptual approval. @lhoestq have you tested it with datasets to check loading time is faster?

[lhoestq]

It's been in prod in datasets-viewer and it fixes the HfHubHTTPError (Too Many Requests) both for the FineWeb's viewer and also for loading the mmteb datasets in datasets

[Wauplin]

Great, thanks for confirming 👍

[Wauplin]

@lhoestq hot-fix released in 0.23.1: https://github.com/huggingface/huggingface_hub/releases/tag/v0.23.1

[lhoestq]

thank you !!