Document to compress data files before uploading

[albertvillanova]

In our docs to Share a dataset to the Hub, we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the .giattributes file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to compress their data files (using ZIP for example) before uploading them to the Hub.

• Compressed files are tracked by Git LFS in our default .gitattributes file

What do you think?
CC: @stevhliu

See related issue:

• https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1

[stevhliu]

Great idea!

Should we also take this opportunity to include some audio/image file formats? Currently, it still reads very text heavy. Something like:

We support many text, audio, and image data extensions such as .zip, .rar, .mp3, and .jpg among many others. For data extensions like .csv, .json, .jsonl, and txt, we recommend compressing them before uploading to the Hub. These file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the .gitattributes file in your repository for a complete list of supported file extensions.

[albertvillanova]

Hi @stevhliu, thanks for your suggestion.

I agree it is a good opportunity to mention that audio/image file formats are also supported.

Nit:
I would not mention .zip, .rar after "text, audio, and image data extensions". Those are "compression" extensions and not "text, audio, and image data extensions".

What about something similar to:

We support many text, audio, and image data extensions such as .csv, .mp3, and .jpg among many others. For text data extensions like .csv, .json, .jsonl, and .txt, we recommend compressing them before uploading to the Hub (to .zip or .gz file extension for example).

Note that text file extensions are not tracked by Git LFS by default, and if they're too large, they will not be committed and uploaded. Take a look at the .gitattributes file in your repository for a complete list of tracked file extensions by default.

Note that for compressions I have mentioned:

• gz, to compress individual files
• zip, to compress and archive multiple files; zip is preferred rather than tar because it supports streaming out of the box

[stevhliu]

Perfect, thanks for making the distinction between compression and data extensions!