Neural Network for Digit Recognition

This is a fun beginner project creating a basic neural network from scratch, by only using Math and NumPy. The Neural Network is trained on the popular MNIST dataset which contains gray-scale images of hand-drawn digits, with each 784 pixels (28x28). The training model utilizes a simple architecture with ReLU and softmax activation functions, predicting the correct digit based on the pixel values and continuously improving its accuracy through iterative training.

Key Finding: The accuracy of the handwritten digit prediction on the training data (83.75%) is lower than on the data tested (84.75%)! This indicates that there're meaningful differences in the datasets (e.g. the test set is probably less diverse and therefore easier for the model to predict).

Table of Contents

• Introduction
• Data Source
• Architecture
• Algorithm
• Results

Introduction

This is my first hands-on beginner project attempting to write a code for a basic neural network recognizing handwritten digits.
Through the past months, I've been getting more and more interested in machine learning and how neural networks work, so I decided it was about time to try it out myself.

I started by watching a YouTube video to get familiar with the math and code that I needed. Then I opened my JupyterLab notebook, imported the training data, preprocessed it, splitting the data into a training and validation set, I shuffeled it and finally normalized the pixel values so they would be between 0 and 1. After that followed the forward and backward propagation, optimization of the parameters and finally I applied the gradient descent algorithm to make predictions. At the end I imported a different test data, letting it run through the model once again to see how well my neural network was doing giving me a new accuracy of predictions.
Additionally, I added many comments to each step in order to help me (and maybe other beginners) understand what each funtion does, so I wouldn't get confused.

Unfortunately, the initial code I used from the video, presented some errors (even after making sure that I corrected the ones from the video itself). This made it definitely more complicated than expected but very realistic as I understand now that writing code is not always a piece of cake. ;)
For the very last section (cell [15]; I forgot to mention this in my kernel) I partially used ChatGPT 3.5 to help me solve what's wrong, making my code run smoothly after hours of work.

I'm very happy with the outcome and hope by uploading this file, I can help viewers with understanding how this neural network runs and inspire others to try it out themselves!

Data Source

I found both of my MNIST datasets on Kaggle, each consisting of hand-drawn gray-scale images ranging from 0-9. Both sets are extremely clean and don't need any modification. Here is more information about the data, taken directly from Kaggle .

The data files train.csv and test.csv contain gray-scale images of hand-drawn digits, from zero through nine.
Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255, inclusive.
The training data set, (train.csv), has 785 columns. The first column, called "label", is the digit that was drawn by the user. The rest of the columns contain the pixel-values of the associated image.

Train Data: https://www.kaggle.com/competitions/digit-recognizer/data?select=train.csv
Test Data: https://www.kaggle.com/datasets/oddrationale/mnist-in-csv?select=mnist_test.csv

Architecture

The neural network's architecture includes an input layer, two hidden layers, and one output layer.
The input layer consists of 784 neurons corresponding to each pixel in the image. Each of the two hidden layers contains 10 neurons, both using the ReLU activation function, while the output layer employs the softmax activation function.

1. Rectified Linear Unit
The ReLU activation function is used in the hidden layers to introduce non-linearity to the network by replacing negative values with zero. This helps the network capture complex patterns in the data more effectively by removing linearity constraints.


2. Softmax
The softmax function transforms the raw output values from the previous layer into a probability distribution over multiple classes. This allows us to interpret the output as the likelihood of each class being the correct prediction. Softmax ensures that the sum of all probabilities adds up to 1, making it suitable for multi-class classification tasks, like digit recognition.

Algorithm

The model is trained using gradient descent optimization. During training, the network parameters (weights and biases) are initialized randomly and updated iteratively to minimize the loss function, which measures the difference between predicted and actual digit labels. Backpropagation is utilized to compute gradients efficiently, allowing for the adjustment of parameters to improve model performance.

Results

After training, the model's performance is evaluated on a new, separate test dataset. The accuracy of the neural network on the new data is 84.75%, which is pretty good for such a basic model.

As previously mentioned, the accuracy of the predicted initial training data is 83.75% which is a little lower than the accuracy on the test data set. This is not very common but suggests that both datasets are to some degree different from another. The training data might be more complex and diverse whereas the test data might present a lack in those features.

In conclusion this hands-on beginner project was a lot of fun for me and I'm inspired by the insights that I have gained and the progress I've made in understanding neural networks and machine learning principles, to keep practicing and dive deeper into advanced concepts in order to be able to tackle more complex machine learning challenges in the future.

Thanks for checking out my project!