ML MNIST Challenge

This contest was offered within TU Munich's course Machine Learning (IN2064).
The goal was to implement k-NN, Neural Network, Logistic Regression and Gaussian Process Classifier in python from scratch and achieve minimal average test error among these classifiers on well-known MNIST dataset, without ensemble learning.

Results

Algorithm	Description	Test Error, %
k-NN	3-NN, Euclidean distance, uniform weights. Preprocessing: Feature vectors extracted from NN.	1.13
k-NN2	3-NN, Euclidean distance, uniform weights. Preprocessing: Augment (training) data (×9) by using random rotations, shifts, Gaussian blur and dropout pixels; PCA-35 whitening and multiplying each feature vector by e11.6 · s, where s – normalized explained variance by the respective principal axis. (equivalent to applying PCA whitening with accordingly weighted Euclidean distance.	2.06
NN	MLP 784-1337-D(0.05)-911-D(0.1)-666-333-128-10 (D – dropout); hidden activations – LeakyReLU(0.01), output – softmax; loss – categorical cross-entropy; 1024 batches; 42 epochs; optimizer – Adam (learning rate 5 · 10–5, rest – defaults from paper). Preprocessing: Augment (training) data (×5) by using random rotations, shifts, Gaussian blur.	1.04
LogReg	32 batches; 91 epoch; L2-penalty, λ = 3.16 · 10–4; optimizer – Adam (learning rate 10–3, rest – defaults from paper) Preprocessing: Feature vectors extracted from NN.	1.01
GPC	794 random data points were used for training; σn = 0; RBF kernel (σf = 0.4217, γ = 1/2l2 = 0.0008511); Newton iterations for Laplace approximation till ΔLog-Marginal-Likelihood ≤ 10–7; solve linear systems iteratively using CG with 10–7 tolerance; for prediction generate 2000 samples for each test point. Preprocessing: Feature vectors extracted from NN.	1.59

Also check for some plots (confusion matrices, learning curves etc.) in experiments/plots/.

How to install

git clone https://github.com/monsta-hd/ml-mnist
cd ml-mnist/
sudo pip install -r requirements.txt

After installation, tests can be run with:

make test

How to run

Check main.py to reproduce training and testing the final models.
Check also experiments notebook to see what I've tried.

System

All computations and time measurements were made on laptop i7-5500U CPU @ 2.40GHz x 4 12GB RAM

Future work

Here the list of what can also be tried regarding these particular 4 ML algorithms (didn't have time to check it, or it was forbidden by the rules, e.g. ensemble learning):

• Model averaging for k-NN: train a group of k-NNs with different parameter k (say, 2, 4, ..., 128) and average their predictions;
• More sophisticated metrics (say, from scipy.spatial.distance) for k-NN;
• Weighting metrics according to some other functions of explained variance from PCA;
• NCA;
• Different kernels or compound kernels for k-NN;
• Commitee of MLPs, CNN, commitee of CNNs or more advanced NNs;
• Unsupervised pretraining for MLP/CNN;
• Different kernels or compound kernels for GPCs;
• 10 one-vs-rest GPCs;
• Use derivatives of Log-Marginal-Likelihood for multiclass Laplace approximation w.r.t kernel parameters for more efficient gradient-based optimization;
• Model averaging for GPCs: train a collection of GPCs on different parts of the data and then average their predictions (or bagging);
• IVM.