Datacomp dataset download

[mactavish91]

Hello,

I am attempting to download the complete datacomp dataset but have encountered significant difficulties. As of now, more than a third of the links I’ve attempted to access are invalid. I’m curious to know how others have successfully downloaded the entire dataset. Could you share the method you used? I greatly appreciate any guidance and look forward to your response.

[Beckschen]

Hello, and thank you for expressing your interest!

I share similar concerns as you do. We downloaded the dataset around May 2023, achieving a download success rate of approximately 95% to 97%. I am not sure what is the success rate for now.
The Datacomp-1B dataset card can be also found here: https://huggingface.co/datasets/mlfoundations/datacomp_1b

Best
Jieneng

[Beckschen]

There are similar issues for your references:

mlfoundations/datacomp#68

mlfoundations/datacomp#39

mlfoundations/datacomp#41