Downloading MNIST dataset with torchvision gives HTTP Error 403

[Borda]

🐛 Bug

I'm getting a 403 error when I try to download MNIST dataset with torchvision 0.4.2.

To Reproduce

../.local/lib/python3.6/site-packages/torchvision/datasets/mnist.py:68: in __init__
    self.download()
../.local/lib/python3.6/site-packages/torchvision/datasets/mnist.py:135: in download
    download_and_extract_archive(url, download_root=self.raw_folder, filename=filename)
../.local/lib/python3.6/site-packages/torchvision/datasets/utils.py:248: in download_and_extract_archive
    download_url(url, download_root, filename, md5)
../.local/lib/python3.6/site-packages/torchvision/datasets/utils.py:96: in download_url
    raise e
../.local/lib/python3.6/site-packages/torchvision/datasets/utils.py:84: in download_url
    reporthook=gen_bar_updater()
/usr/local/lib/python3.6/urllib/request.py:248: in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
/usr/local/lib/python3.6/urllib/request.py:223: in urlopen
    return opener.open(url, data, timeout)
/usr/local/lib/python3.6/urllib/request.py:532: in open
    response = meth(req, response)
/usr/local/lib/python3.6/urllib/request.py:642: in http_response
    'http', request, response, code, msg, hdrs)
/usr/local/lib/python3.6/urllib/request.py:570: in error
    return self._call_chain(*args)
/usr/local/lib/python3.6/urllib/request.py:504: in _call_chain
    result = func(*args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <urllib.request.HTTPDefaultErrorHandler object at 0x7efbf9edaac8>
req = <urllib.request.Request object at 0x7efbf9eda8d0>
fp = <http.client.HTTPResponse object at 0x7efbf9edaf98>, code = 403
msg = 'Forbidden', hdrs = <http.client.HTTPMessage object at 0x7efbf9ea22b0>

    def http_error_default(self, req, fp, code, msg, hdrs):
>       raise HTTPError(req.full_url, code, msg, hdrs, fp)
E       urllib.error.HTTPError: HTTP Error 403: Forbidden

Environment

• torch==1.3.1
• torchvision==0.4.2

Additional context

https://app.circleci.com/jobs/github/PyTorchLightning/pytorch-lightning/6877

[fmassa]

Thanks for reporting! I can reproduce the issue locally, and downloading from the browser works.

I don't yet know what the root cause is though.

[fmassa]

I think we might need to pass header in the download_url function

vision/torchvision/datasets/utils.py

Lines 59 to 100 in c3e2b01

	def download_url(url, root, filename=None, md5=None):
	"""Download a file from a url and place it in root.
	Args:
	url (str): URL to download file from
	root (str): Directory to place downloaded file in
	filename (str, optional): Name to save the file under. If None, use the basename of the URL
	md5 (str, optional): MD5 checksum of the download. If None, do not check
	"""
	from six.moves import urllib
	root = os.path.expanduser(root)
	if not filename:
	filename = os.path.basename(url)
	fpath = os.path.join(root, filename)
	makedir_exist_ok(root)
	# check if file is already present locally
	if check_integrity(fpath, md5):
	print('Using downloaded and verified file: ' + fpath)
	else: # download the file
	try:
	print('Downloading ' + url + ' to ' + fpath)
	urllib.request.urlretrieve(
	url, fpath,
	reporthook=gen_bar_updater()
	)
	except (urllib.error.URLError, IOError) as e:
	if url[:5] == 'https':
	url = url.replace('https:', 'http:')
	print('Failed download. Trying https -> http instead.'
	' Downloading ' + url + ' to ' + fpath)
	urllib.request.urlretrieve(
	url, fpath,
	reporthook=gen_bar_updater()
	)
	else:
	raise e
	# check integrity of downloaded file
	if not check_integrity(fpath, md5):
	raise RuntimeError("File not found or corrupted.")

according to https://stackoverflow.com/questions/13303449/urllib2-httperror-http-error-403-forbidden

cc @cpuhrsch @vincentqb @zhangguanheng66 for awareness

[soumith]

this is because the download links for mnist at https://github.com/pytorch/vision/blob/master/torchvision/datasets/mnist.py#L33-L36 are hosted on yann.lecun.com and that server has moved under CloudFlare protection.

@fmassa we need to maybe mirror and change the URLs to maybe the PyTorch S3 bucket or something

[Borda]

so could we make a hot-fix somehow?

[fmassa]

@Borda I haven't tried the current hotfix I mentioned, but I think it might be possible, would you be able to try it and send a PR? Otherwise I'll look into it early next week (I'm working towards ECCV deadline tomorrow)

And I would rather avoid hosting the datasets ourselves, as this would give precedence on us storing the datasets.

[eduardo4jesus]

Is there any way to have a quick fix without using the master?
I am concerned about the potential changes I have to do in my code for going from the version I am using (1.4.0) and the master.

[mvelebit]

@eduardo4jesus You can explicitly add headers as stated above, something alike:

opener = urllib.request.URLopener()
opener.addheader('User-Agent', some_user_agent)
opener.retrieve(
    url, fpath,
    reporthook=gen_bar_updater()
)

(line 81 and onwards in vision/torchvision/datasets/utils.py). Seems to be a quick workaround that works.