Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MNIST downloader can't handle corrupt local files? #4588

Closed
AlexDBlack opened this issue Feb 1, 2018 · 2 comments

Comments

@AlexDBlack
Copy link
Contributor

commented Feb 1, 2018

Files to reproduce:
MNIST_corrupt.zip

Once a local (incomplete download) corruption occurs, it seems the downloader can't handle this...
i.e., same exception, no matter how many times the test is run.

o.d.b.MnistFetcher - Downloading MNIST...

java.io.EOFException: Unexpected end of ZLIB input stream

	at java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:240)
	at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
	at java.io.FilterInputStream.read(FilterInputStream.java:107)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792)
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
	at org.deeplearning4j.util.ArchiveUtils.unzipFileTo(ArchiveUtils.java:147)
	at org.deeplearning4j.base.MnistFetcher.downloadAndUntar(MnistFetcher.java:159)
	at org.deeplearning4j.base.MnistFetcherTest.testMnistDataFetcher(MnistFetcherTest.java:15)

@AlexDBlack AlexDBlack added the Bug label Feb 1, 2018

@AlexDBlack AlexDBlack self-assigned this Feb 7, 2018

@AlexDBlack

This comment has been minimized.

Copy link
Contributor Author

commented Feb 7, 2018

Actually, it's even worse: it's also possible to get into a state where local MNIST files are corrupt, and features end up as all 0s... :(

AlexDBlack added a commit that referenced this issue Feb 7, 2018
AlexDBlack added a commit that referenced this issue Feb 7, 2018
Merge pull request #4625 from deeplearning4j/ab_4588_mnist
#4588 Add checksum to MNIST local files, re-download if required; extra test
@lock

This comment has been minimized.

Copy link

commented Sep 23, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Sep 23, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
1 participant
You can’t perform that action at this time.