An notebook that implements the backpropagation algorithm from scratch to train a Multi-Layer Perceptron (MLP) with a single hidden layer.
This exercise provides a practical, in-depth look at the backpropagation algorithm, the cornerstone of training neural networks. We will build a simple MLP and train it on various data patterns to develop an intuition for the training dynamics.
The main goal is to observe and understand the convergence of the network's weights and biases. We will systematically experiment with key hyperparameters to see their direct impact on the learning process. 🧠
- Implement Backpropagation: Code the forward and backward passes of a neural network to update its weights.
- Analyze Weight Convergence: Visualize how the network's weights change over time and see how quickly or slowly they converge to a solution.
- Explore Hyperparameter Effects: Investigate the interplay between crucial training parameters:
- Learning Rate: See how it affects the speed and stability of convergence.
- Number of Iterations: Understand the trade-off between training time and model performance.
- Number of Data Points: Observe how dataset size influences the final learned pattern.
- Multi-Layer Perceptron (MLP): The architecture of our neural network, consisting of an input layer, a single hidden layer, and an output layer.
- Feedforward Pass: The process of passing input data through the network to generate an output.
- Backpropagation Algorithm: The method of propagating the error backward through the network to calculate the gradients needed for weight updates.
- Gradient Descent: The optimization algorithm used to update the weights based on the calculated gradients.
- Loss Function: A function (e.g., Mean Squared Error) that quantifies the difference between the predicted output and the actual target.