improve error handling for GDrive downloads

[pmeier]

We have plenty of reports that the download of datasets hosted on GDrive does not work as expected:

• Failed to download CelebA dataset using download=True #1920
• Unable to load CelebA dataset. File is not zip file error.  #2262
• Failed to download CelebA in Colab #2317
• Bad zipfile error while downloading CelebA dataset #2756
• Unable to load CelebA dataset: "File is not zip file" error. #3708
• WIDERFace download fails with zip error #4615
• zipfile.BadZipFile exception when downloading WIDERFace (traced to download_file_from_google_drive function) #5615
• Caltech101, Caltech256 downloads are broken due to Google Drive redirect "scan for viruses" popup #5716
• PCAM dataset download fails #5800

Although most of them are closed, they see another comment from time to time since a user stumbled upon "the same issue"

The errors non-descriptive in most cases. The problem is that we don't check the MD5 sum after the download and naively write the response from GDrive to disk. In contrast, on download_url we perform such a check

vision/torchvision/datasets/utils.py

Lines 150 to 152 in aa21197

	# check integrity of downloaded file
	if not check_integrity(fpath, md5):
	raise RuntimeError("File not found or corrupted.")

The most common failure case is GDrive returning an unknown API response as HTML. This PR adds a MD5 check after the download with an additional HTML check to make the error message more descriptive.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit a6cdc84 (more details on the Dr. CI page):

---

💚 💚 Looks good so far! There are no failures yet. 💚 💚

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[NicolasHug]

Thanks @pmeier .

The most common failure case is GDrive returning an unknown API response as HTML

Shouldn't we check the HTML code of the download / reply from GDrive instead of downloading then?

[pmeier]

Shouldn't we check the HTML code of the download / reply from GDrive instead of downloading then?

Unfortunately, GDrive mostly just returns 200 and thus there is very little we can do without actually inspecting the response. Maybe we can improve this by always analyzing the first chunk of data rather than only if we detect an MD5 mismatch. Thoughts?

[NicolasHug]

analyzing the first chunk of data rather than only if we detect an MD5 mismatch. Thoughts

If you mean checking against the HTML regex right after the files are downloaded, then yeah I feel like this might be where the check needs to be

[NicolasHug]

Thanks Philip