Skip to content
Evaluation of fashion-MNIST with a simple cnn
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
images Move / rename images Jan 4, 2018
models/simple_cnn Move / rename images Jan 4, 2018
Extracting_MNIST.ipynb Adding example for extracting MNIST Dec 28, 2017
README.md Add usage Jan 4, 2018
Simple_Convolutional_Neural_Network_Fashion-MNIST.ipynb small changes for generalization Jan 9, 2018

README.md

Image classification on fashion-MNIST

I would like to share my results (93.43% accuracy on average) on the fashion-MNIST dataset. You can find further informations about the dataset on Zalando Research and Kaggle This dataset is a great option instead of using traditional handwritten MNIST.
Thanks to Han and Kashif!

Evaluation procedure

I splitted the training data randomly in train (80%) and validation (20%). The testset of 10k images are used for final evaluation. I created 5 models with the same architecture but with random train/validation data. I only saved the weights of every model with best loss. Finally I used the models to evaluate them on the testset. The average loss/accuracy of the 5 models is the final result.
Furthermore I evaluate the generalization of the network for classifying traditional MNIST at the end.

Results

Actually I'm focussed on a very simple architecture with less than 500,000 parameters which could run on CPU with 4GB memory. If you are interested in further results, you can find them on Zalando benchmark. If it fits in time, I will evaluate architectures e.g. DenseNet.

I'm just using two convolutional layers, batchnorm, dropout and three fully connected layers. For a detailed implementation check the model definition in the jupyter notebook.
This is the summary of the current model:

It's easy to reach an accuracy over 90% but around 92% it stucks very fast. If we take a look on the confusion matrix, we will find out that distinguishing between Shirt and T-Shirt/top is very difficult:

Here's an example where my model fails. I'm not a fashion expert. Maybe thats the reason why I don't see any difference. If you could explain why this is a shirt and not a t-shirt I would appreciate it:

Another two plots which illustrate the accuracy / loss of training and validation over the time:

And a detailed plot of iteration 0:

Scores for training: 0.1066 loss / 95.99% accuracy.
Scores for validation: 0.1245 loss / 95.64% accuracy.
Scores for test: 0.2149 loss / 93.43% accuracy.

If you like, you can download the saved models/weights and history for fashion-MNIST.

MNIST

I used the same architecture of neural network to train on traditional MNIST.
Scores for training: 0.078 loss / 99.74% accuracy.
Scores for validation: 0.161 loss / 99.61% accuracy.
Scores for test: 0.0248 loss / 99.43% accuracy.

In the plot below you can track training / validation for MNIST:

If you like, you can download the saved models/weights and history for MNIST.

Overview

Fashion-MNIST
test accuracy
Fashion-MNIST
train accuracy
Fashion-MNIST
validation accuracy
MNIST
test accuracy
Add. Settings Trainable
Params
93.43% 95.99% 95.64% 99.43% BatchSize : 250
Epochs : 80
Data augmentation (2x)
493,772

Requirements

Usage

Feel free to use this code and models for improving on your own. I would appreciate it if you give any feedback.

The easiest way to use the code examples is to download / clone the whole repository. The only thing that’s missing is the folder data with the given subfolder structure for the input images:

data
  ├── test
  |   ├── fashion_mnist
  |   |   ├── 0
  |   |   ├── 1
  |   |   ├── ...
  |   |
  |   ├── mnist
  |   |   ├── 0
  |   |   ├── 1
  |   |   ├── ...
  |
  ├── train
  |   ├── fashion_mnist
  |   |   ├── 0
  |   |   ├── 1
  |   |   ├── ...
  |   |
  |   ├── mnist
  |   |   ├── 0
  |   |   ├── 1
  |   |   ├── ...

For getting this structure automatically I wrote a extractor finding in this jupyter notebook. This extracts the images of a MNIST IDX file.

If you have any questions or hints contact me through an issue. Thanks!

References

[1] Fashion-MNIST
[2] Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

Author

Christopher Masch

You can’t perform that action at this time.