A feedforward neural network for binary classification, implemented from scratch with numpy - no autograd, no framework. Supports arbitrary layer sizes, relu/tanh/sigmoid activations, L2 regularization, and gradient checking to verify the backprop derivation is actually correct.
For layer i, with weights W_i, bias b_i, and activation g:
Z_i = W_i . A_(i-1) + b_i
A_i = g(Z_i)
A_0 = X. The final layer uses sigmoid, so A_L is a per-example probability.
Loss - binary cross-entropy with L2 regularization:
cost = -(1/m) * sum( Y*log(A_L) + (1-Y)*log(1-A_L) ) + (lambda/2m) * sum(W_i^2)
Backprop - starting from dA_L = -(Y/A_L - (1-Y)/(1-A_L)) (the BCE derivative), each layer going backward computes:
dZ_i = dA_i * g'(Z_i)
dW_i = (1/m) * dZ_i . A_(i-1)^T + (lambda/m) * W_i
db_i = (1/m) * sum(dZ_i, axis=1)
dA_(i-1) = W_i^T . dZ_i
which is the standard chain-rule derivation for this architecture, gradient-checked below rather than taken on faith.
pip install numpy matplotlib
python NeuralNetwork.py
This trains on data_banknote_authentication.csv (banknote authentication, nearly linearly separable), runs gradient_check() - which compares the analytical gradients above to numerical gradients from centered finite differences - and asserts on both. Non-zero exit on failure.
Latest run:
gradient check relative error: 1.58e-09
test accuracy: 1.0000
all checks passed
ANN.ipynb has the same model applied to the banknote and occupancy-detection datasets with pandas/sklearn for data handling (not required by the model itself).