A mirror of the CelebA dataset on S3, similar to the one for MNIST

[lucaslingle]

🚀 Feature

A mirror of the CelebA dataset on S3, to support reliable downloading of the dataset.

Motivation

Downloading the CelebA dataset through TorchVision is currently an issue, due to a common quota exceeded error associated with Google Drive, where the dataset owners currently host the dataset. To resolve a similar issue with MNIST, a mirror was set up on S3. As I type this, this mirror is the active source of the download for that dataset, as Yann LeCun's website is currently down.

Pitch

I would like the TorchVision developers to contact the CelebA authors, and request permission to host the dataset on S3. Reliable access to this dataset will increase its usage and popularity further, so I don't see why they would object. The dataset can then be hosted in the same S3 bucket as MNIST, and the source code for torchvision.datasets.CelebA can be modified to point to this mirror if the primary download source fails to deliver the dataset in pristine condition.

Alternatives

Alternative solutions include future prospective torchvision users waiting an indefinite amount of time to download the dataset on their local machines, thus slowing down developer productivity.

Additional context

I also opened a bug ticket for this issue.

cc @pmeier

[pmeier]

Hey @lucaslingle I like the idea, but I wouldn't get my hopes up. We got permission from Yann LeCun, because he is a FB employee and PyTorch is a FB product. In general the following from our README holds:

This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets [...].

MNIST is kind of an outlier, because it is still the most used dataset in tutorials. So if the download breaks we get multiple issues per hour asking to fix it. I don't want to say that CelebA is a less important dataset, but if it breaks it is easier to manage for us.

Cc: @fmassa @seemethere

[lucaslingle]

I am very sympathetic to this argument, and of course it is up to you guys.

It may provide some useful context to know that the dataset is still unavailable from the original source, for the same reason as when I originally posted: quota issues. If this persists for the remainder of the year, it seems likely that this dataset will die out, unless someone saves it for the sake of "open source science".

[lucaslingle]

I finally got the dataset. Although I continue to support this ticket, I suppose we can close it if you're certain there's no chance it'll happen. You don't need to keep the issue open on my behalf.

Thanks for your consideration!

[pmeier]

The problem I see with all of this is, that it shifts the maintenance burden not only to us but to every utility library that provides wrappers for datasets (for example tensorflow/datasets/issues/3150).

If this persists for the remainder of the year, it seems likely that this dataset will die out, unless someone saves it for the sake of "open source science".

Of course this is bad for reproducibility, but if the authors do not want or cannot put in the effort maybe it is ok to let it die out.

If you are worried about that happening, I suggest you take this to the authors themselves. Maybe they give others permissions to host the dataset. If we get to that point, we can discuss this again. Until then, from the torchvision side we will not pursue that. #3710 should finally give a expressive error message so users encountering this have a course of action.

Thus, I will close this for now.