New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added support for the QMNIST dataset #995
Conversation
Note: With python-2.7, there is no lzma module by default. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for the PR Leon!
I've made a few minor comments. I believe we can fix CI by moving the import of lzma
inside the function call.
Let me know if you want me to address the comments that I've made.
Codecov Report
@@ Coverage Diff @@
## master #995 +/- ##
==========================================
- Coverage 62.69% 62.54% -0.15%
==========================================
Files 65 65
Lines 5080 5148 +68
Branches 761 772 +11
==========================================
+ Hits 3185 3220 +35
- Misses 1678 1711 +33
Partials 217 217
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for the PR Leon!
QMNIST https://github.com/facebookresearch/qmnist
Patch
The patch has two commits.
read_sn3_pascalvincent_tensor
that reads all variants of the mnist file format. Can read compressed files. Can accept a file object instead of a filename. Redefine functionsread_label_file
andread_image_file
using this generic function.qmnist.py
loader provided with the QMNIST distribution. An interesting aspect is the argument 'what' that specifies which subset to access: 'train', 'test', 'test10k', 'test60k', 'nist'. When argumentwhat
is absent, the 'train' argument is consulted.Remarks.
The way I handle compressed files in
read_sn3_pascalvincent_tensor
does not square well with the new utility functiondownload_and_extract_archive
. My take is that this function should only deal with archives (zip, tar, tgz) but not compressed files (gz, xz). In the end, the QMNIST class usesdownload_url
instead.Instead the
QMNIST
class usesdownload_url
. My take is that the