Fix (Fashion|K)MNIST download and MNIST download test #3557
Conversation
| return collect_download_configs(lambda: datasets.MNIST(ROOT, download=True), name="MNIST") | ||
| with unittest.mock.patch.object(datasets.MNIST, "mirrors", datasets.MNIST.mirrors[-1:]): | ||
| return collect_download_configs(lambda: datasets.MNIST(ROOT, download=True), name="MNIST") |
There was a problem hiding this comment.
This is a hacky fix, but the test infrastructure is currently not able to handle mirrors. During test collection, our tests mock out download_url and record all calls to for each dataset. Afterwards we do head requests against the logged URLs to check if they are accessible. Since we can't know if a request returns an error during collection, the mirror URLs are never collected.
Maybe we should add a mirror functionality to download_url that handles this internally. If we had this, we could add a functionality like this to our tests that warns in case a mirror is down.
| mirrors = [ | ||
| "http://fashion-mnist.s3-website.eu-central-1.amazonaws.com/" | ||
| ] |
There was a problem hiding this comment.
(Fashion|K)MNIST use the same download logic as MNIST. Thus, we need to also provide the URLs in the same fashion (no pun intended 😉) although we only have a single mirror.
|
Only |
* add mirrors to (Fashion|K)MNIST * fix download tests for MNIST
* datasets: Fallback to our own mirrors for mnist (#3544) We are experiencing 403s when trying to download from the main mnist site so lets fallback to our own mirror on failure. Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Co-authored-by: Francisco Massa <fvsmassa@gmail.com> * Fix (Fashion|K)MNIST download and MNIST download test (#3557) * add mirrors to (Fashion|K)MNIST * fix download tests for MNIST Co-authored-by: Eli Uriegas <1700823+seemethere@users.noreply.github.com> Co-authored-by: Francisco Massa <fvsmassa@gmail.com>
Summary: * add mirrors to (Fashion|K)MNIST * fix download tests for MNIST Reviewed By: fmassa Differential Revision: D27128007 fbshipit-source-id: 2ca4bba7d8e823cdca3174f408f14e9e6e2e8346
Fixes #3556.