A from-scratch implementation of several core machine learning models — including a Perceptron, a regression neural network, a digit classifier, and a language identifier — built on top of a custom neural network mini-library.
This project implements four progressively complex machine learning models, each solving a distinct problem. Rather than using a high-level framework like PyTorch or TensorFlow, the project is built on a custom-designed computation graph library (nn.py) that manually handles forward passes, backpropagation, and gradient descent. This makes the inner workings of neural networks fully transparent and explicit.
- Language: Python 3
- Core Library: NumPy (for all matrix operations and numerical computation)
- Custom Framework: A hand-built neural network library (
nn.py) implementing a computation graph with automatic differentiation (backpropagation) - Datasets: MNIST (handwritten digits,
.npz) and a multi-language word dataset (lang_id.npz) - Key ML Concepts Applied:
- Perceptron learning rule
- Feedforward neural networks (MLPs)
- Recurrent neural networks (RNNs)
- ReLU activations
- Softmax cross-entropy loss
- Mean squared error (square loss)
- Mini-batch gradient descent
- Validation-based early stopping
machinelearning/
├── nn.py # Custom neural network mini-library (computation graph)
├── models.py # All four ML model implementations
├── backend.py # Dataset loading, iteration, and validation utilities
├── autograder.py # Test suite for grading each model
└── data/
├── mnist.npz # MNIST handwritten digit dataset
└── lang_id.npz # Multi-language word dataset
nn.py is a custom-built automatic differentiation library. It defines a computation graph of Node objects and implements both forward computation and backpropagation from scratch using NumPy.
Node types implemented:
| Node | Purpose |
|---|---|
Parameter |
Learnable weight/bias matrices, initialized with uniform random values scaled by layer size |
Constant |
Immutable nodes for inputs, labels, and computed gradients |
Add |
Element-wise matrix addition |
AddBias |
Broadcasts and adds a bias vector to a batch of feature vectors |
Linear |
Matrix multiplication (linear transformation): features @ weights |
DotProduct |
Batched dot product between a feature vector and a weight row |
ReLU |
Element-wise activation: max(x, 0) |
SquareLoss |
Mean squared error: mean(0.5 * (a - b)^2) |
SoftmaxLoss |
Numerically stable batched softmax cross-entropy loss |
nn.gradients(loss, parameters) performs reverse-mode automatic differentiation (backpropagation) by traversing the computation graph in reverse topological order and accumulating gradients for each Parameter.
A classic binary linear classifier that learns to separate data into two classes (+1 or −1).
- Architecture: A single weight vector
wof shape(1 × dimensions) - Scoring:
score(x) = dot(x, w) - Prediction: Returns
+1if the score is ≥ 0, otherwise−1 - Training: Iterates over the dataset with batch size 1; when a misclassification occurs, updates weights by
w += y * x(the perceptron update rule with learning rate 1.0). Training repeats until a full pass over the data produces zero mistakes (convergence).
To test:
python autograder.py -q q1A deep feedforward neural network that approximates smooth non-linear functions — trained to fit sin(x) over the interval [−2π, 2π].
- Architecture: 4-layer MLP:
1 → 256 → 256 → 256 → 1- Hidden layers use ReLU activations
- Output layer is linear (no activation), producing a continuous scalar prediction
- Loss: Mean squared error (
nn.SquareLoss) - Optimizer: Mini-batch gradient descent
- Batch size: 200
- Learning rate: 0.01
- Stopping criterion: Training loss ≤ 0.02 (or up to 6,000 epochs)
To test:
python autograder.py -q q2A multi-layer perceptron (MLP) that classifies handwritten digits from the MNIST dataset.
- Input: 28×28 grayscale images flattened to 784-dimensional vectors
- Output: A 10-dimensional logit vector — one score per digit class (0–9)
- Architecture:
784 → 200 → 100 → 10- Two hidden layers with ReLU activations
- Linear output layer (no final activation — raw logits fed directly into softmax loss)
- Loss: Softmax cross-entropy (
nn.SoftmaxLoss) - Optimizer: Mini-batch gradient descent
- Batch size: 100
- Learning rate: 0.2
- Stopping criterion: Validation accuracy ≥ 97.5% (or up to 30 epochs)
To test:
python autograder.py -q q3A Recurrent Neural Network (RNN) that identifies which language a word belongs to, processing one character at a time.
- Languages: English, Spanish, Finnish, Dutch, Polish (5 classes)
- Input: Each word is represented as a sequence of one-hot character vectors. The combined alphabet across all five languages contains 47 unique characters, so each character is a vector of shape
(batch_size × 47). - Architecture: Vanilla RNN with the recurrence:
After processing all characters, the final hidden state is projected to 5 class logits:
h_t = ReLU(x_t @ Wx + h_{t-1} @ Wh + bh)logits = h_final @ Wo + bo- Hidden size: 128
- Parameters:
Wx(47×128),Wh(128×128),bh(1×128),Wo(128×5),bo(1×5)
- Loss: Softmax cross-entropy (
nn.SoftmaxLoss) - Optimizer: Mini-batch gradient descent
- Batch size: 100
- Learning rate: 0.1
- Stopping criterion: Validation accuracy ≥ 82% (or up to 50 epochs)
To test:
python autograder.py -q q4All models can be tested individually or all at once using the autograder:
# Test all questions
python autograder.py
# Test a specific model
python autograder.py -q q1 # Perceptron
python autograder.py -q q2 # Regression
python autograder.py -q q3 # Digit Classification
python autograder.py -q q4 # Language IdentificationAll commands should be run from inside the
machinelearning/directory.